You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Ziliang Guo (JIRA)" <ji...@apache.org> on 2009/08/26 20:49:59 UTC
[jira] Created: (HDFS-573) Porting libhdfs to Windows
Porting libhdfs to Windows
--------------------------
Key: HDFS-573
URL: https://issues.apache.org/jira/browse/HDFS-573
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Environment: Windows, Visual Studio 2008
Reporter: Ziliang Guo
The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748092#action_12748092 ]
dhruba borthakur commented on HDFS-573:
---------------------------------------
Is there a way to create a new file that contains finctions that implements hsearch/hcreate etc for Windows? in that case, we can continue to use the current code on Linux.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "Ziliang Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12757411#action_12757411 ]
Ziliang Guo commented on HDFS-573:
----------------------------------
Faisal and I both work for Condor so what he actually did was run the tests
on the stuff we have sitting in the repo, not some generated patch. I'll be
generating that patch hopefully next week, as school has started for me and
I had to deal with that this week.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748105#action_12748105 ]
dhruba borthakur commented on HDFS-573:
---------------------------------------
OK, thanks for the explanation.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12756554#action_12756554 ]
dhruba borthakur commented on HDFS-573:
---------------------------------------
> ran unit tests on Ziliang's patch for libhdfs
where is the patch file?
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "Ziliang Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749550#action_12749550 ]
Ziliang Guo commented on HDFS-573:
----------------------------------
Fasial has tested the current code on Linux and besides one minor fix it
seems to work the same as before, so at least nothing seems to have broken
during the reworking. Considering how extensive the changes were, do you
guys want a diff or just a copy of the current source code as is?
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "Ziliang Guo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748099#action_12748099 ]
Ziliang Guo commented on HDFS-573:
----------------------------------
I should have been more explicit. That is exactly what I have done for
hcreate/hsearch. The header with the functions is only included if WIN32 is
defined. I also modified the locking macros in a similar #ifdef statement
to the appropriate Windows locks. The current code should continue to work
on Linux, barring a screwup in my moving the variable declarations to the
top of the function as required in C89. However, there was a lot of moving
things around.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "Faisal Khan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755610#action_12755610 ]
Faisal Khan commented on HDFS-573:
----------------------------------
I ran unit tests on Ziliang's patch for libhdfs on Linux and here is the output http://pages.cs.wisc.edu/~faisal/libhdfs_testresult.txt . Tests look ok.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-573) Porting libhdfs to Windows
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749671#action_12749671 ]
dhruba borthakur commented on HDFS-573:
---------------------------------------
Some of the details are in http://wiki.apache.org/hadoop/HowToContribute.
You have to create a "svn diff" in your workspace and then attaching it as a text file to this JIRA. Also, please see if you can run the libhdfs unit test.
> Porting libhdfs to Windows
> --------------------------
>
> Key: HDFS-573
> URL: https://issues.apache.org/jira/browse/HDFS-573
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Environment: Windows, Visual Studio 2008
> Reporter: Ziliang Guo
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> The current C code in libhdfs is written using C99 conventions and also uses a few POSIX specific functions such as hcreate, hsearch, and pthread mutex locks. To compile it using Visual Studio would require a conversion of the code in hdfsJniHelper.c and hdfs.c to C89 and replacement/reimplementation of the POSIX functions. The code also uses the stdint.h header, which is not part of the original C89, but there exists what appears to be a BSD licensed reimplementation written to be compatible with MSVC floating around. I have already done the other necessary conversions, as well as created a simplistic hash bucket for use with hcreate and hsearch and successfully built a DLL of libhdfs. Further testing is needed to see if it is usable by other programs to actually access hdfs, which will likely happen in the next few weeks as the Condor Project continues with its file transfer work.
> In the process, I've removed a few what I believe are extraneous consts and also fixed an incorrect array initialization where someone was attempting to initialize with something like this: JavaVMOption options[noArgs]; where noArgs was being incremented in the code above. This was in the hdfsJniHelper.c file, in the getJNIEnv function.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.