You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Lucas Stanley <lu...@gmail.com> on 2013/06/11 03:57:50 UTC
SSD support in HDFS
Hi,
Is it possible to tell Apache HDFS to store some files on SSD and the rest
of the files on spinning disks?
So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
directory in HDFS to put all files in that dir on the SSD?
I think Intel's Hadoop distribution is working on some SSD support right?
Re: SSD support in HDFS
Posted by Alexey Babutin <zo...@gmail.com>.
You can use BTIER <http://www.lessfs.com> in linux.(I don't know its
stable or not).With FreeBSD you can use ZFS +L2ARC on SSD.
On Thu, Jun 13, 2013 at 6:35 AM, Michael Segel <mi...@hotmail.com>wrote:
> I could have sworn there was a thread on this already. (Maybe the HBase
> list?)
>
> Andrew P. kinda nailed it when he talked about the fact that you had to
> write the replication(s).
>
> If you wanted improved performance, why not look at the hybrid drives that
> have a small SSD buffer and a spinning disk?
>
> I don't know but it may be what you're looking for.
>
> HTH
>
> -Mike
>
> On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
>
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
>
>> Hi Lucas,
>>
>> HDFS does not have this capability right now, but there has been some
>> preliminary discussion around adding features to support it. You might
>> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
>> notifications about the discussion.
>>
>> https://issues.apache.org/jira/browse/HDFS-2832
>> https://issues.apache.org/jira/browse/HDFS-4672
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Is it possible to tell Apache HDFS to store some files on SSD and the
>>> rest of the files on spinning disks?
>>>
>>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>>> directory in HDFS to put all files in that dir on the SSD?
>>>
>>> I think Intel's Hadoop distribution is working on some SSD support right?
>>>
>>
>>
>
>
Re: SSD support in HDFS
Posted by Alexey Babutin <zo...@gmail.com>.
You can use BTIER <http://www.lessfs.com> in linux.(I don't know its
stable or not).With FreeBSD you can use ZFS +L2ARC on SSD.
On Thu, Jun 13, 2013 at 6:35 AM, Michael Segel <mi...@hotmail.com>wrote:
> I could have sworn there was a thread on this already. (Maybe the HBase
> list?)
>
> Andrew P. kinda nailed it when he talked about the fact that you had to
> write the replication(s).
>
> If you wanted improved performance, why not look at the hybrid drives that
> have a small SSD buffer and a spinning disk?
>
> I don't know but it may be what you're looking for.
>
> HTH
>
> -Mike
>
> On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
>
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
>
>> Hi Lucas,
>>
>> HDFS does not have this capability right now, but there has been some
>> preliminary discussion around adding features to support it. You might
>> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
>> notifications about the discussion.
>>
>> https://issues.apache.org/jira/browse/HDFS-2832
>> https://issues.apache.org/jira/browse/HDFS-4672
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Is it possible to tell Apache HDFS to store some files on SSD and the
>>> rest of the files on spinning disks?
>>>
>>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>>> directory in HDFS to put all files in that dir on the SSD?
>>>
>>> I think Intel's Hadoop distribution is working on some SSD support right?
>>>
>>
>>
>
>
Re: SSD support in HDFS
Posted by Alexey Babutin <zo...@gmail.com>.
You can use BTIER <http://www.lessfs.com> in linux.(I don't know its
stable or not).With FreeBSD you can use ZFS +L2ARC on SSD.
On Thu, Jun 13, 2013 at 6:35 AM, Michael Segel <mi...@hotmail.com>wrote:
> I could have sworn there was a thread on this already. (Maybe the HBase
> list?)
>
> Andrew P. kinda nailed it when he talked about the fact that you had to
> write the replication(s).
>
> If you wanted improved performance, why not look at the hybrid drives that
> have a small SSD buffer and a spinning disk?
>
> I don't know but it may be what you're looking for.
>
> HTH
>
> -Mike
>
> On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
>
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
>
>> Hi Lucas,
>>
>> HDFS does not have this capability right now, but there has been some
>> preliminary discussion around adding features to support it. You might
>> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
>> notifications about the discussion.
>>
>> https://issues.apache.org/jira/browse/HDFS-2832
>> https://issues.apache.org/jira/browse/HDFS-4672
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Is it possible to tell Apache HDFS to store some files on SSD and the
>>> rest of the files on spinning disks?
>>>
>>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>>> directory in HDFS to put all files in that dir on the SSD?
>>>
>>> I think Intel's Hadoop distribution is working on some SSD support right?
>>>
>>
>>
>
>
Re: SSD support in HDFS
Posted by Alexey Babutin <zo...@gmail.com>.
You can use BTIER <http://www.lessfs.com> in linux.(I don't know its
stable or not).With FreeBSD you can use ZFS +L2ARC on SSD.
On Thu, Jun 13, 2013 at 6:35 AM, Michael Segel <mi...@hotmail.com>wrote:
> I could have sworn there was a thread on this already. (Maybe the HBase
> list?)
>
> Andrew P. kinda nailed it when he talked about the fact that you had to
> write the replication(s).
>
> If you wanted improved performance, why not look at the hybrid drives that
> have a small SSD buffer and a spinning disk?
>
> I don't know but it may be what you're looking for.
>
> HTH
>
> -Mike
>
> On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
>
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
>
>> Hi Lucas,
>>
>> HDFS does not have this capability right now, but there has been some
>> preliminary discussion around adding features to support it. You might
>> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
>> notifications about the discussion.
>>
>> https://issues.apache.org/jira/browse/HDFS-2832
>> https://issues.apache.org/jira/browse/HDFS-4672
>>
>> Chris Nauroth
>> Hortonworks
>> http://hortonworks.com/
>>
>>
>>
>> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> Is it possible to tell Apache HDFS to store some files on SSD and the
>>> rest of the files on spinning disks?
>>>
>>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>>> directory in HDFS to put all files in that dir on the SSD?
>>>
>>> I think Intel's Hadoop distribution is working on some SSD support right?
>>>
>>
>>
>
>
Re: SSD support in HDFS
Posted by Michael Segel <mi...@hotmail.com>.
I could have sworn there was a thread on this already. (Maybe the HBase list?)
Andrew P. kinda nailed it when he talked about the fact that you had to write the replication(s).
If you wanted improved performance, why not look at the hybrid drives that have a small SSD buffer and a spinning disk?
I don't know but it may be what you're looking for.
HTH
-Mike
On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some preliminary discussion around adding features to support it. You might want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
>
Re: SSD support in HDFS
Posted by Michael Segel <mi...@hotmail.com>.
I could have sworn there was a thread on this already. (Maybe the HBase list?)
Andrew P. kinda nailed it when he talked about the fact that you had to write the replication(s).
If you wanted improved performance, why not look at the hybrid drives that have a small SSD buffer and a spinning disk?
I don't know but it may be what you're looking for.
HTH
-Mike
On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some preliminary discussion around adding features to support it. You might want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
>
Re: SSD support in HDFS
Posted by Michael Segel <mi...@hotmail.com>.
I could have sworn there was a thread on this already. (Maybe the HBase list?)
Andrew P. kinda nailed it when he talked about the fact that you had to write the replication(s).
If you wanted improved performance, why not look at the hybrid drives that have a small SSD buffer and a spinning disk?
I don't know but it may be what you're looking for.
HTH
-Mike
On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some preliminary discussion around adding features to support it. You might want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
>
Re: SSD support in HDFS
Posted by Michael Segel <mi...@hotmail.com>.
I could have sworn there was a thread on this already. (Maybe the HBase list?)
Andrew P. kinda nailed it when he talked about the fact that you had to write the replication(s).
If you wanted improved performance, why not look at the hybrid drives that have a small SSD buffer and a spinning disk?
I don't know but it may be what you're looking for.
HTH
-Mike
On Jun 12, 2013, at 5:18 AM, Lucas Stanley <lu...@gmail.com> wrote:
> Thanks Chris and Phil.
>
>
> On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com> wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some preliminary discussion around adding features to support it. You might want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
>
Re: SSD support in HDFS
Posted by Lucas Stanley <lu...@gmail.com>.
Thanks Chris and Phil.
On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some
> preliminary discussion around adding features to support it. You might
> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
> notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>
>> Hi,
>>
>> Is it possible to tell Apache HDFS to store some files on SSD and the
>> rest of the files on spinning disks?
>>
>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>> directory in HDFS to put all files in that dir on the SSD?
>>
>> I think Intel's Hadoop distribution is working on some SSD support right?
>>
>
>
Re: SSD support in HDFS
Posted by Lucas Stanley <lu...@gmail.com>.
Thanks Chris and Phil.
On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some
> preliminary discussion around adding features to support it. You might
> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
> notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>
>> Hi,
>>
>> Is it possible to tell Apache HDFS to store some files on SSD and the
>> rest of the files on spinning disks?
>>
>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>> directory in HDFS to put all files in that dir on the SSD?
>>
>> I think Intel's Hadoop distribution is working on some SSD support right?
>>
>
>
Re: SSD support in HDFS
Posted by Lucas Stanley <lu...@gmail.com>.
Thanks Chris and Phil.
On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some
> preliminary discussion around adding features to support it. You might
> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
> notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>
>> Hi,
>>
>> Is it possible to tell Apache HDFS to store some files on SSD and the
>> rest of the files on spinning disks?
>>
>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>> directory in HDFS to put all files in that dir on the SSD?
>>
>> I think Intel's Hadoop distribution is working on some SSD support right?
>>
>
>
Re: SSD support in HDFS
Posted by Lucas Stanley <lu...@gmail.com>.
Thanks Chris and Phil.
On Tue, Jun 11, 2013 at 1:31 PM, Chris Nauroth <cn...@hortonworks.com>wrote:
> Hi Lucas,
>
> HDFS does not have this capability right now, but there has been some
> preliminary discussion around adding features to support it. You might
> want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
> notifications about the discussion.
>
> https://issues.apache.org/jira/browse/HDFS-2832
> https://issues.apache.org/jira/browse/HDFS-4672
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
>
> On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com>wrote:
>
>> Hi,
>>
>> Is it possible to tell Apache HDFS to store some files on SSD and the
>> rest of the files on spinning disks?
>>
>> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
>> directory in HDFS to put all files in that dir on the SSD?
>>
>> I think Intel's Hadoop distribution is working on some SSD support right?
>>
>
>
Re: SSD support in HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Lucas,
HDFS does not have this capability right now, but there has been some
preliminary discussion around adding features to support it. You might
want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
notifications about the discussion.
https://issues.apache.org/jira/browse/HDFS-2832
https://issues.apache.org/jira/browse/HDFS-4672
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest
> of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
> directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
Re: SSD support in HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Lucas,
HDFS does not have this capability right now, but there has been some
preliminary discussion around adding features to support it. You might
want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
notifications about the discussion.
https://issues.apache.org/jira/browse/HDFS-2832
https://issues.apache.org/jira/browse/HDFS-4672
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest
> of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
> directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
Re: SSD support in HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Lucas,
HDFS does not have this capability right now, but there has been some
preliminary discussion around adding features to support it. You might
want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
notifications about the discussion.
https://issues.apache.org/jira/browse/HDFS-2832
https://issues.apache.org/jira/browse/HDFS-4672
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest
> of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
> directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>
Re: SSD support in HDFS
Posted by Chris Nauroth <cn...@hortonworks.com>.
Hi Lucas,
HDFS does not have this capability right now, but there has been some
preliminary discussion around adding features to support it. You might
want to follow jira issues HDFS-2832 and HDFS-4672 if you'd like to receive
notifications about the discussion.
https://issues.apache.org/jira/browse/HDFS-2832
https://issues.apache.org/jira/browse/HDFS-4672
Chris Nauroth
Hortonworks
http://hortonworks.com/
On Mon, Jun 10, 2013 at 6:57 PM, Lucas Stanley <lu...@gmail.com> wrote:
> Hi,
>
> Is it possible to tell Apache HDFS to store some files on SSD and the rest
> of the files on spinning disks?
>
> So if each on my nodes has 1 SSD and 5 spinning disks, can I configure a
> directory in HDFS to put all files in that dir on the SSD?
>
> I think Intel's Hadoop distribution is working on some SSD support right?
>