You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Tim Miller (JIRA)" <ji...@apache.org> on 2012/06/29 23:03:44 UTC
[jira] [Created] (HADOOP-8545) filesystem implementation for
OpenStack Swift
Tim Miller created HADOOP-8545:
----------------------------------
Summary: filesystem implementation for OpenStack Swift
Key: HADOOP-8545
URL: https://issues.apache.org/jira/browse/HADOOP-8545
Project: Hadoop Common
Issue Type: New Feature
Components: fs
Reporter: Tim Miller
Priority: Minor
Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8545) Filesystem Implementation for
OpenStack Swift
Posted by "Nirmal Ranganathan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510062#comment-13510062 ]
Nirmal Ranganathan commented on HADOOP-8545:
--------------------------------------------
Setting up a local swift install for testing.
In a clean vm run the following
git clone https://github.com/openstack-dev/devstack
cd devstack
echo "ENABLED_SERVICES=key,swift
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data" > localrc
./stack.sh
This will pull down the latest swift code.. install dependencies and setup all the services required for swift.
Note: this is just a dev setup.
To test:
source openrc
swift --help
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>
> Key: HADOOP-8545
> URL: https://issues.apache.org/jira/browse/HADOOP-8545
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Reporter: Tim Miller
> Priority: Minor
>
> Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for
OpenStack Swift
Posted by "Tim Miller (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Miller updated HADOOP-8545:
-------------------------------
Summary: Filesystem Implementation for OpenStack Swift (was: filesystem implementation for OpenStack Swift)
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>
> Key: HADOOP-8545
> URL: https://issues.apache.org/jira/browse/HADOOP-8545
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Reporter: Tim Miller
> Priority: Minor
>
> Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for
OpenStack Swift
Posted by "David Dobbins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Dobbins updated HADOOP-8545:
----------------------------------
Attachment: HADOOP-8545-1.patch
Here is another patch that contains a filesystem implementation for Swift. It currently relies on the Rackspace java_cloudfiles api to communicate with Swift.
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>
> Key: HADOOP-8545
> URL: https://issues.apache.org/jira/browse/HADOOP-8545
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 0.22.0
> Reporter: Tim Miller
> Priority: Minor
> Attachments: HADOOP-8545-1.patch, HADOOP-8545.patch
>
>
> Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for
OpenStack Swift
Posted by "Dmitry Mezhensky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitry Mezhensky updated HADOOP-8545:
-------------------------------------
Affects Version/s: 0.22.0
Release Note:
Added file system implementation for OpenStack Swift.
There are two implementation: block and native (similar to Amazon S3 integration).
Data locality issue solved by patch in Swift, commit procedure to OpenStack is in progress.
To use implementation add to core-site.xml following:
...
<property>
<name>fs.swift.impl</name>
<value>com.mirantis.fs.SwiftFileSystem</value>
</property>
<property>
<name>fs.swift.block.impl</name>
<value>com.mirantis.fs.block.SwiftBlockFileSystem</value>
</property>
...
In MapReduce job specify following configs for OpenStack Keystone authentication:
conf.set("swift.auth.url", "http://172.18.66.117:5000/v2.0/tokens");
conf.set("swift.tenant", "superuser");
conf.set("swift.username", "admin1");
conf.set("swift.password", "password");
conf.setInt("swift.http.port", 8080);
conf.setInt("swift.https.port", 443);
Additional information specified on github: https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration
Status: Patch Available (was: Open)
Please review coding, tests.
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>
> Key: HADOOP-8545
> URL: https://issues.apache.org/jira/browse/HADOOP-8545
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 0.22.0
> Reporter: Tim Miller
> Priority: Minor
> Attachments: HADOOP-8545.patch
>
>
> Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8545) Filesystem Implementation for
OpenStack Swift
Posted by "Dmitry Mezhensky (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-8545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmitry Mezhensky updated HADOOP-8545:
-------------------------------------
Attachment: HADOOP-8545.patch
> Filesystem Implementation for OpenStack Swift
> ---------------------------------------------
>
> Key: HADOOP-8545
> URL: https://issues.apache.org/jira/browse/HADOOP-8545
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs
> Affects Versions: 0.22.0
> Reporter: Tim Miller
> Priority: Minor
> Attachments: HADOOP-8545.patch
>
>
> Add a filesystem implementation for OpenStack Swift object store, similar to the one which exists today for S3.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira