You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/04/07 16:34:47 UTC
[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #782:
HDDS-3352. Support for native ozone filesystem client using libhdfs.
aryangupta1998 opened a new pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782
## What changes were proposed in this pull request?
This jira is to bring in support for native ozone filesystem client using libhdfs.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3352
Please replace this section with the link to the Apache JIRA)
## How was this patch tested?
Tested manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407472539
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
+ snprintf(string, len + 3, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
Added #define O3FS "o3fs://"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 closed pull request #782: HDDS-3352.
Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 closed pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407391314
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
+ snprintf(string, len + 3, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
lets have o3fs as a #define at the start of the file.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on issue #782: HDDS-3352.
Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-610976586
> Wow, nice work.
>
> "127.0.0.1" seems to be hardcoded (but I am not native in C, I might missed sg)
>
> And I got some ideas immediately about integrating it with the build and creating tests, etc. But they can be done in follow-up PRs: good to see the first steps...
@elek Thanks!
Earlier host-name, bucket-name, etc were hardcoded. I have made some changes, now we can pass host-name, port-number, bucket-name and volume-name as parameters through command line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #782:
HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
fapifta commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r405505521
##########
File path: hadoop-ozone/native-client/README.md
##########
@@ -0,0 +1,79 @@
+#Overview
+
+libozfs is a JNI based C API for Ozone File System. It provides with read and write functionality on OzoneFileSystem. It also uses some functions from HDFS(Hadoop Distributed File System) for which it uses libhdfs, which is a JNI based C API for Hadoop’s Distributed File System (HDFS). It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and the filesystem. libhdfs is part of the Hadoop distribution and comes pre-compiled in $HADOOP_HDFS_HOME/lib/native/libhdfs.so .
+
+#The APIs
+
+The libozfs APIs are a subset of the Ozone FileSystem APIs. The header file for libozfs describes each API in detail and is available in $HADOOP_HDFS_HOME/include/ozfs.h.
+
+#Requirements:
+
+1.Hadoop with compiled libhdfs.so
+2.Linux kernel > 2.6.9 with fuse, which is the default or Fuse 2.7.x, 2.8.x installed. See: http://fuse.sourceforge.net/ or even easier: http://dag.wieers.com/rpm/packages/fuse
Review comment:
Why we need fuse for the current code?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on issue #782: HDDS-3352.
Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-612894239
> I feel we should rename the filesystem to o3 in place of oz. I mean lets call it libo3fs and o3fs_read and write and every where else ?
I have renamed all the files(o3fs_read, write, etc) and directories(libo3fs, libo3fs-examples)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 edited a comment on issue #782:
HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 edited a comment on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-610976586
> Wow, nice work.
>
> "127.0.0.1" seems to be hardcoded (but I am not native in C, I might missed sg)
>
> And I got some ideas immediately about integrating it with the build and creating tests, etc. But they can be done in follow-up PRs: good to see the first steps...
@elek Thanks!
Earlier host-name, bucket-name, etc were hardcoded. I have made some changes, now we can pass host-name, port-number, bucket-name and volume-name as parameters through command line.
Also I will try to develop a new pom.xml so that we can compile this using maven.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407471348
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
Review comment:
Added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407390426
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
+ snprintf(string, len + 3, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
Can you please add an examle here ? I mean how does the final URI look like ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r405551309
##########
File path: hadoop-ozone/native-client/README.md
##########
@@ -0,0 +1,79 @@
+#Overview
+
+libozfs is a JNI based C API for Ozone File System. It provides with read and write functionality on OzoneFileSystem. It also uses some functions from HDFS(Hadoop Distributed File System) for which it uses libhdfs, which is a JNI based C API for Hadoop’s Distributed File System (HDFS). It provides C APIs to a subset of the HDFS APIs to manipulate HDFS files and the filesystem. libhdfs is part of the Hadoop distribution and comes pre-compiled in $HADOOP_HDFS_HOME/lib/native/libhdfs.so .
+
+#The APIs
+
+The libozfs APIs are a subset of the Ozone FileSystem APIs. The header file for libozfs describes each API in detail and is available in $HADOOP_HDFS_HOME/include/ozfs.h.
+
+#Requirements:
+
+1.Hadoop with compiled libhdfs.so
+2.Linux kernel > 2.6.9 with fuse, which is the default or Fuse 2.7.x, 2.8.x installed. See: http://fuse.sourceforge.net/ or even easier: http://dag.wieers.com/rpm/packages/fuse
Review comment:
Sorry thats a mistake, actually I’m working on FUSE also thats why by mistake I added. I have removed it as we don’t need fuse for this implementation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407391833
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
Review comment:
why len +2 here and len+3 on the next line ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 opened a new pull request #782:
HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 opened a new pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782
## What changes were proposed in this pull request?
This jira is to bring in support for native ozone filesystem client using libhdfs.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3352
## How was this patch tested?
Tested manually
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on issue #782: HDDS-3352. Support
for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
fapifta commented on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-614304228
Thank you for addressing the concerns, and taking in the suggestions. From my side I am +1 for this change. (non-binding)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on issue #782: HDDS-3352. Support
for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
fapifta commented on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-610943026
Simple and elegant contribution, thank you for it.
I think the 127.0.0.1 is not that big of a deal, it is in an example code, and pretty obvious, so that might suffice here.
If you can address the concern I have added inline, and we get to a solution on that one, I am +1 for this change, I might add some small things to the readme, I have not gone through it yet, but will do shortly ;)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407391573
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
Review comment:
lets have { and } after the if and at the end of if statement
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] fapifta commented on a change in pull request #782:
HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
fapifta commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r405500452
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,42 @@
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ char string[100] = "";
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ snprintf(string, len + 5, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
Thank you for the contribution, I have one question here.
The doc of snprintf as I understand says, it writes the output to the first param which is a char*, it writes at most the number of characters specified in the second argument, and the text is specified from the third argument on just as it should be defined for printf.
About the char* that serves as a buffer in this case the doc says "The buffer should have a size of at least n characters." where n is the number specified in the second parameter.
If these are correct, and as string is defined as a char[100], if the length of host, bucket, vol, plus the lenght of "o3fs://", plus 3 for the two dot and the null terminator is larger than 100, then as this call tries to write this many characters to the char array, it will index out from the array which will most likely cause an error.
If this is true (I am not really familiar with C, but you can try this easily with a longer bucket or volume or host name), then we need to create the char array with a sufficient size to store len+5 characters.
And one more thing... I am not sure why the second character of snprintf here is len+5, I see that there are two extra dots in the host, but that does not explain 5. Can you explain this a bit better in the code, with like naming it and extracting it to a local variable, or by adding a comment, that would help to understand that a bit easier.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407471088
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
+ snprintf(string, len + 3, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
Example added
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] mukul1987 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
mukul1987 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407390873
##########
File path: hadoop-ozone/native-client/libozone/ozfs.h
##########
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef OZFS_DOT_H
+#define OZFS_DOT_H
+
+#include "hdfs/hdfs.h"
+
+struct hdfs_internal;
+typedef struct hdfs_internal* ozfsFS;
+
+struct hdfsFile_internal;
+typedef struct hdfsFile_internal* ozfsFile;
+
+ozfsFS ozfsConnect(const char* nn, tPort port, const char* bucket, const char* volume);
+
+ozfsFile ozfsOpenFile(ozfsFS fs, const char *path, int flags, int bufferSize, short replication, tSize blockSize);
Review comment:
More than 80 characters in a line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on issue #782: HDDS-3352. Support for
native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
elek commented on issue #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#issuecomment-610856023
Wow, nice work.
"127.0.0.1" seems to be hardcoded (but I am not native in C, I might missed sg)
And I got some ideas immediately about integrating it with the build and creating tests, etc. But they can be done in follow-up PRs: good to see the first steps...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407471222
##########
File path: hadoop-ozone/native-client/libozone/ozfs.h
##########
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#ifndef OZFS_DOT_H
+#define OZFS_DOT_H
+
+#include "hdfs/hdfs.h"
+
+struct hdfs_internal;
+typedef struct hdfs_internal* ozfsFS;
+
+struct hdfsFile_internal;
+typedef struct hdfsFile_internal* ozfsFile;
+
+ozfsFS ozfsConnect(const char* nn, tPort port, const char* bucket, const char* volume);
+
+ozfsFile ozfsOpenFile(ozfsFS fs, const char *path, int flags, int bufferSize, short replication, tSize blockSize);
Review comment:
Resolved.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r405549703
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,42 @@
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ char string[100] = "";
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ snprintf(string, len + 5, "o3fs://%s.%s.%s", bucket, vol, host);
Review comment:
Thanks for pointing out @fapifta
I changed len+5 to len+3 (len+2 for the two dots and +1 for the null character that’s being appended by snprintf).
Also I changed string[100] to string[len+2].
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] aryangupta1998 commented on a change in pull request
#782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
Posted by GitBox <gi...@apache.org>.
aryangupta1998 commented on a change in pull request #782: HDDS-3352. Support for native ozone filesystem client using libhdfs.
URL: https://github.com/apache/hadoop-ozone/pull/782#discussion_r407472321
##########
File path: hadoop-ozone/native-client/libozone/ozfs.c
##########
@@ -0,0 +1,60 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "ozfs.h"
+#include "hdfs/hdfs.h"
+#include <fcntl.h>
+#include <inttypes.h>
+#include <stdio.h>
+#include <string.h>
+
+
+ozfsFS ozfsConnect(const char *host, tPort port, const char *bucket, const char *vol)
+{
+ struct hdfsBuilder *bld = hdfsNewBuilder();
+ int len = 0;
+ if (!bld)
+ return NULL;
+ len = strlen(host) + strlen(bucket) + strlen(vol) + strlen("o3fs://");
+ char string[len + 2];
Review comment:
snprintf just appends a null character at the end and we dont need that null character thats why i mentioned string[len+2]. In snprintf we have to pass len+3 because its function requirement.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org