You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Wei-Chiu Chuang <we...@cloudera.com.INVALID> on 2019/06/10 17:08:03 UTC

[DISCUSS] HDFS roadmap/wish list

Hi!

I am soliciting feedbacks for HDFS roadmap items and wish list in the
future Hadoop releases. A community meetup
<https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
is happening soon, and perhaps we can use this thread to converge on things
we should talk about there.

I am aware of several major features that merged into trunk, such as RBF,
Consistent Standby Serving Reads, as well as some recent features that
merged into 3.2.0 release (storage policy satisfier).

What else should we be doing? I have a laundry list of supportability
improvement projects, mostly about improving performance or making
performance diagnostics easier. I can share the list if folks are
interested.

Are there things we should do to make developer's life easier or things
that would be nice to have for downstream applications? I know Sahil Takiar
made a series of improvements in HDFS for Impala recently, and those
improvements are applicable to other downstreamers such as HBase. Or would
it help if we provide more Hadoop API examples?

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Jeff,

Would Hadoop encryption zone/Transparent Data Encryption (TDE)
<https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html>
address this use case? Files within encryption zone are encrypted
transparently. Data is encrypted on DataNodes and are decrypted at client
side. Or would Data Transfer Encryption
<https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SecureMode.html#Data_Encryption_on_Block_data_transfer.>
work for you? These are pretty mature these days so probably worth trying.

Definitely let me know if the encryption system I mentioned above doesn't
work for you. I know there are assumptions behind the design and doesn't
work for all use cases (it doesn't support per column encryption key in
HBase)

On Mon, Jun 10, 2019 at 8:07 PM Jeff Hubbs <jh...@att.net> wrote:

> Hi, Wei-Chiu -
>
> I don't know if this is something already in the pipeline for 3.x, but I'd
> like to see a mechanism in HDFS that encrypts blocks pre-storage such that
> I'd only have to manage keys in one place (NameManager?). If that
> capability existed, then I could move blocks around an unsafe network
> and/or not have to worry about my worker nodes having volume-level or
> whole-disk-level encryption. Even if I have Hadoop traffic only crossing a
> LAN that's captive to the cluster, I might still have to worry about worker
> nodes being stolen outright or having the drive(s) taken out of them.
>
> - Jeff
>
> On 6/10/19 8:40 PM, Wei-Chiu Chuang wrote:
>
>
> Thank you Sudeep for the feedback,
>
> To be more specific, what sort of examples are??you looking for???
>
> On another note, I had written some docs of extended length about Hadoop
> code base and internal designs. I should probably make those public to
> share the knowledge (or fix my grammar errors, for that matter)
>
> On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur <
> sudeepthakur90@gmail.com> wrote:
>
>> Hi ,
>>
>> Examples are most helpful for developer. Please add examples as much as
>> we can.
>>
>> Thanks??
>> Sudeep Thakur
>>
>> On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang
>> <we...@cloudera.com.invalid> <we...@cloudera.com.invalid> wrote:
>>
>>> Hi!
>>>
>>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>>> future Hadoop releases. A community meetup
>>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>>> is happening soon, and perhaps we can use this thread to converge on things
>>> we should talk about there.
>>>
>>> I am aware of several major features that merged into trunk, such as
>>> RBF, Consistent Standby Serving Reads, as well as some recent features that
>>> merged into 3.2.0 release (storage policy satisfier).
>>>
>>> What else should we be doing? I have a laundry list of supportability
>>> improvement projects, mostly about improving performance or making
>>> performance diagnostics easier. I can share the list if folks are
>>> interested.
>>>
>>> Are there things we should do to make developer's life easier or things
>>> that would be nice to have for downstream applications? I know??Sahil
>>> Takiar made a series of improvements in HDFS for Impala recently, and those
>>> improvements are applicable to other downstreamers such as HBase. Or would
>>> it help if we provide more Hadoop API examples???
>>>
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Jeff Hubbs <jh...@att.net>.
Hi, Wei-Chiu -

I don't know if this is something already in the pipeline for 3.x, but 
I'd like to see a mechanism in HDFS that encrypts blocks pre-storage 
such that I'd only have to manage keys in one place (NameManager?). If 
that capability existed, then I could move blocks around an unsafe 
network and/or not have to worry about my worker nodes having 
volume-level or whole-disk-level encryption. Even if I have Hadoop 
traffic only crossing a LAN that's captive to the cluster, I might still 
have to worry about worker nodes being stolen outright or having the 
drive(s) taken out of them.

- Jeff

On 6/10/19 8:40 PM, Wei-Chiu Chuang wrote:


> Thank you Sudeep for the feedback,
>
> To be more specific, what sort of examples are??you looking for?
>
> On another note, I had written some docs of extended length about 
> Hadoop code base and internal designs. I should probably make those 
> public to share the knowledge (or fix my grammar errors, for that matter)
>
> On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur 
> <sudeepthakur90@gmail.com <ma...@gmail.com>> wrote:
>
>     Hi ,
>
>     Examples are most helpful for developer. Please add examples as
>     much as we can.
>
>     Thanks
>     Sudeep Thakur
>
>     On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang
>     <we...@cloudera.com.invalid> wrote:
>
>         Hi!
>
>         I am soliciting feedbacks for HDFS roadmap items and wish list
>         in the future Hadoop releases. A community meetup
>         <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>         is happening soon, and perhaps we can use this thread to
>         converge on things we should talk about there.
>
>         I am aware of several major features that merged into trunk,
>         such as RBF, Consistent Standby Serving Reads, as well as some
>         recent features that merged into 3.2.0 release (storage policy
>         satisfier).
>
>         What else should we be doing? I have a laundry list of
>         supportability improvement projects, mostly about improving
>         performance or making performance diagnostics easier. I can
>         share the list if folks are interested.
>
>         Are there things we should do to make developer's life easier
>         or things that would be nice to have for downstream
>         applications? I know??Sahil Takiar made a series of
>         improvements in HDFS for Impala recently, and those
>         improvements are applicable to other downstreamers such as
>         HBase. Or would it help if we provide more Hadoop API examples?
>


Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thank you Sudeep for the feedback,

To be more specific, what sort of examples are you looking for?

On another note, I had written some docs of extended length about Hadoop
code base and internal designs. I should probably make those public to
share the knowledge (or fix my grammar errors, for that matter)

On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur <
sudeepthakur90@gmail.com> wrote:

> Hi ,
>
> Examples are most helpful for developer. Please add examples as much as we
> can.
>
> Thanks
> Sudeep Thakur
>
> On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang
> <we...@cloudera.com.invalid> wrote:
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thank you Sudeep for the feedback,

To be more specific, what sort of examples are you looking for?

On another note, I had written some docs of extended length about Hadoop
code base and internal designs. I should probably make those public to
share the knowledge (or fix my grammar errors, for that matter)

On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur <
sudeepthakur90@gmail.com> wrote:

> Hi ,
>
> Examples are most helpful for developer. Please add examples as much as we
> can.
>
> Thanks
> Sudeep Thakur
>
> On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang
> <we...@cloudera.com.invalid> wrote:
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Thank you Sudeep for the feedback,

To be more specific, what sort of examples are you looking for?

On another note, I had written some docs of extended length about Hadoop
code base and internal designs. I should probably make those public to
share the knowledge (or fix my grammar errors, for that matter)

On Mon, Jun 10, 2019 at 12:11 PM Sudeep Singh Thakur <
sudeepthakur90@gmail.com> wrote:

> Hi ,
>
> Examples are most helpful for developer. Please add examples as much as we
> can.
>
> Thanks
> Sudeep Thakur
>
> On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang
> <we...@cloudera.com.invalid> wrote:
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Sudeep Singh Thakur <su...@gmail.com>.
Hi ,

Examples are most helpful for developer. Please add examples as much as we
can.

Thanks
Sudeep Thakur

On Mon, Jun 10, 2019, 10:38 PM Wei-Chiu Chuang <we...@cloudera.com.invalid>
wrote:

> Hi!
>
> I am soliciting feedbacks for HDFS roadmap items and wish list in the
> future Hadoop releases. A community meetup
> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
> is happening soon, and perhaps we can use this thread to converge on things
> we should talk about there.
>
> I am aware of several major features that merged into trunk, such as RBF,
> Consistent Standby Serving Reads, as well as some recent features that
> merged into 3.2.0 release (storage policy satisfier).
>
> What else should we be doing? I have a laundry list of supportability
> improvement projects, mostly about improving performance or making
> performance diagnostics easier. I can share the list if folks are
> interested.
>
> Are there things we should do to make developer's life easier or things
> that would be nice to have for downstream applications? I know Sahil Takiar
> made a series of improvements in HDFS for Impala recently, and those
> improvements are applicable to other downstreamers such as HBase. Or would
> it help if we provide more Hadoop API examples?
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Julien Laurenceau <ju...@pepitedata.com>.
Hi,
Thanks for taking this into account.
As you seems to be interested in adapting the "per directory" HDFS
attributes regarding replication factor, I would also suggest to add a
finer granularity to the DFS blocksize.
The idea would be to optimize the blocksize on a per directory basis. So
that an application dealing with very large files could beneficiate from a
very large block size without impacting everyone.
As I am not familiar with the code, I don't know if this kind of feature
would not broke the simplicity of Hadoop.
Maybe this could be done by adding attributes to the hdfs policies.
Maybe it could also be a good idea to make the replication factor an
attribute of the hdfs policies for coherence sake.

Regards
Julien

Le jeu. 13 juin 2019 à 20:41, Wei-Chiu Chuang <we...@apache.org> a écrit :

> Thank you. I really appreciate your feedback as I don't always know the
> detailed use case for a feature. (For me, it's mostly "hey, this thing is
> broken, fix it")
>
> What are the rest of the community thinks? This is a great opportunity to
> share what you think.
>
> My answers inline:
>
> On Wed, Jun 12, 2019 at 1:12 AM Julien Laurenceau <
> julien.laurenceau@pepitedata.com> wrote:
>
>> Hi,
>>
>> I am not absolutely sure it is not already in a roadmap or supported, but
>> I would appreciate those two features :
>>
>> - First feature : I would also like to be able to use a dedicated
>> directory in HDFS as a /tmp directory leveraging RAMFS for high performing
>> checkpoint of Spark Jobs without using Alluxio or Ignite.
>>
> My current issue is that the RAMFS is only useful with replication factor
>> x1 (in order to avoid network).
>> My default replication factor is x3, but I would need a way to set
>> replication factor x1 on a specific directory (/tmp) for all new writes
>> coming to this directory.
>> Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
>> written.
>> For example, this could be done by specifying the replication factor at
>> the storage policy level.
>> In my view this would dramatically improve the interest of the
>> Lazy-persist storage policy.
>>
>
> I am told LAZY_PERSIST is never considered a completed feature, and two
> Hadoop distros, CDH and HDP don't support it.
>
> But now that I understand the use case, it looks useful now.
>
>> > From the Doc > Note 1: The Lazy_Persist policy is useful only for
>> single replica blocks. For blocks with more than one replicas, all the
>> replicas will be written to DISK since writing only one of the replicas to
>> RAM_DISK does not improve the overall performance.
>> In the current state of HDFS configuration, I only see the following hack
>> (not tested) to implement such a solution : Configure HDFS replication x1
>> as default configuration and use Erasure Coding RS(6,3) for the main
>> storage by attaching an ec storage policy on all directories except /tmp.
>>
>> hdfs ec -setPolicy -path <directory> [-policy <policyName>]
>>
>>
>>
>> - Second feature: a bandwidth throttling dedicated to the re-replication
>> in case of a failed datanode.
>> Something similar to the option dedicated to the balancing algorithm
>> dfs.datanode.balance.bandwidthPerSecbut only for re-replication.
>>
> I am pretty sure I've got people asking about this before a few times.
>
>>
>> Thanks and regards
>> JL
>>
>> Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang
>> <we...@cloudera.com.invalid> a écrit :
>>
>>> Hi!
>>>
>>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>>> future Hadoop releases. A community meetup
>>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>>> is happening soon, and perhaps we can use this thread to converge on things
>>> we should talk about there.
>>>
>>> I am aware of several major features that merged into trunk, such as
>>> RBF, Consistent Standby Serving Reads, as well as some recent features that
>>> merged into 3.2.0 release (storage policy satisfier).
>>>
>>> What else should we be doing? I have a laundry list of supportability
>>> improvement projects, mostly about improving performance or making
>>> performance diagnostics easier. I can share the list if folks are
>>> interested.
>>>
>>> Are there things we should do to make developer's life easier or things
>>> that would be nice to have for downstream applications? I know Sahil Takiar
>>> made a series of improvements in HDFS for Impala recently, and those
>>> improvements are applicable to other downstreamers such as HBase. Or would
>>> it help if we provide more Hadoop API examples?
>>>
>>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@apache.org>.
Thank you. I really appreciate your feedback as I don't always know the
detailed use case for a feature. (For me, it's mostly "hey, this thing is
broken, fix it")

What are the rest of the community thinks? This is a great opportunity to
share what you think.

My answers inline:

On Wed, Jun 12, 2019 at 1:12 AM Julien Laurenceau <
julien.laurenceau@pepitedata.com> wrote:

> Hi,
>
> I am not absolutely sure it is not already in a roadmap or supported, but
> I would appreciate those two features :
>
> - First feature : I would also like to be able to use a dedicated
> directory in HDFS as a /tmp directory leveraging RAMFS for high performing
> checkpoint of Spark Jobs without using Alluxio or Ignite.
>
My current issue is that the RAMFS is only useful with replication factor
> x1 (in order to avoid network).
> My default replication factor is x3, but I would need a way to set
> replication factor x1 on a specific directory (/tmp) for all new writes
> coming to this directory.
> Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
> written.
> For example, this could be done by specifying the replication factor at
> the storage policy level.
> In my view this would dramatically improve the interest of the
> Lazy-persist storage policy.
>

I am told LAZY_PERSIST is never considered a completed feature, and two
Hadoop distros, CDH and HDP don't support it.

But now that I understand the use case, it looks useful now.

> > From the Doc > Note 1: The Lazy_Persist policy is useful only for single
> replica blocks. For blocks with more than one replicas, all the replicas
> will be written to DISK since writing only one of the replicas to RAM_DISK
> does not improve the overall performance.
> In the current state of HDFS configuration, I only see the following hack
> (not tested) to implement such a solution : Configure HDFS replication x1
> as default configuration and use Erasure Coding RS(6,3) for the main
> storage by attaching an ec storage policy on all directories except /tmp.
>
> hdfs ec -setPolicy -path <directory> [-policy <policyName>]
>
>
>
> - Second feature: a bandwidth throttling dedicated to the re-replication
> in case of a failed datanode.
> Something similar to the option dedicated to the balancing algorithm
> dfs.datanode.balance.bandwidthPerSecbut only for re-replication.
>
I am pretty sure I've got people asking about this before a few times.

>
> Thanks and regards
> JL
>
> Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang <we...@cloudera.com.invalid>
> a écrit :
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@apache.org>.
Thank you. I really appreciate your feedback as I don't always know the
detailed use case for a feature. (For me, it's mostly "hey, this thing is
broken, fix it")

What are the rest of the community thinks? This is a great opportunity to
share what you think.

My answers inline:

On Wed, Jun 12, 2019 at 1:12 AM Julien Laurenceau <
julien.laurenceau@pepitedata.com> wrote:

> Hi,
>
> I am not absolutely sure it is not already in a roadmap or supported, but
> I would appreciate those two features :
>
> - First feature : I would also like to be able to use a dedicated
> directory in HDFS as a /tmp directory leveraging RAMFS for high performing
> checkpoint of Spark Jobs without using Alluxio or Ignite.
>
My current issue is that the RAMFS is only useful with replication factor
> x1 (in order to avoid network).
> My default replication factor is x3, but I would need a way to set
> replication factor x1 on a specific directory (/tmp) for all new writes
> coming to this directory.
> Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
> written.
> For example, this could be done by specifying the replication factor at
> the storage policy level.
> In my view this would dramatically improve the interest of the
> Lazy-persist storage policy.
>

I am told LAZY_PERSIST is never considered a completed feature, and two
Hadoop distros, CDH and HDP don't support it.

But now that I understand the use case, it looks useful now.

> > From the Doc > Note 1: The Lazy_Persist policy is useful only for single
> replica blocks. For blocks with more than one replicas, all the replicas
> will be written to DISK since writing only one of the replicas to RAM_DISK
> does not improve the overall performance.
> In the current state of HDFS configuration, I only see the following hack
> (not tested) to implement such a solution : Configure HDFS replication x1
> as default configuration and use Erasure Coding RS(6,3) for the main
> storage by attaching an ec storage policy on all directories except /tmp.
>
> hdfs ec -setPolicy -path <directory> [-policy <policyName>]
>
>
>
> - Second feature: a bandwidth throttling dedicated to the re-replication
> in case of a failed datanode.
> Something similar to the option dedicated to the balancing algorithm
> dfs.datanode.balance.bandwidthPerSecbut only for re-replication.
>
I am pretty sure I've got people asking about this before a few times.

>
> Thanks and regards
> JL
>
> Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang <we...@cloudera.com.invalid>
> a écrit :
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Wei-Chiu Chuang <we...@apache.org>.
Thank you. I really appreciate your feedback as I don't always know the
detailed use case for a feature. (For me, it's mostly "hey, this thing is
broken, fix it")

What are the rest of the community thinks? This is a great opportunity to
share what you think.

My answers inline:

On Wed, Jun 12, 2019 at 1:12 AM Julien Laurenceau <
julien.laurenceau@pepitedata.com> wrote:

> Hi,
>
> I am not absolutely sure it is not already in a roadmap or supported, but
> I would appreciate those two features :
>
> - First feature : I would also like to be able to use a dedicated
> directory in HDFS as a /tmp directory leveraging RAMFS for high performing
> checkpoint of Spark Jobs without using Alluxio or Ignite.
>
My current issue is that the RAMFS is only useful with replication factor
> x1 (in order to avoid network).
> My default replication factor is x3, but I would need a way to set
> replication factor x1 on a specific directory (/tmp) for all new writes
> coming to this directory.
> Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
> written.
> For example, this could be done by specifying the replication factor at
> the storage policy level.
> In my view this would dramatically improve the interest of the
> Lazy-persist storage policy.
>

I am told LAZY_PERSIST is never considered a completed feature, and two
Hadoop distros, CDH and HDP don't support it.

But now that I understand the use case, it looks useful now.

> > From the Doc > Note 1: The Lazy_Persist policy is useful only for single
> replica blocks. For blocks with more than one replicas, all the replicas
> will be written to DISK since writing only one of the replicas to RAM_DISK
> does not improve the overall performance.
> In the current state of HDFS configuration, I only see the following hack
> (not tested) to implement such a solution : Configure HDFS replication x1
> as default configuration and use Erasure Coding RS(6,3) for the main
> storage by attaching an ec storage policy on all directories except /tmp.
>
> hdfs ec -setPolicy -path <directory> [-policy <policyName>]
>
>
>
> - Second feature: a bandwidth throttling dedicated to the re-replication
> in case of a failed datanode.
> Something similar to the option dedicated to the balancing algorithm
> dfs.datanode.balance.bandwidthPerSecbut only for re-replication.
>
I am pretty sure I've got people asking about this before a few times.

>
> Thanks and regards
> JL
>
> Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang <we...@cloudera.com.invalid>
> a écrit :
>
>> Hi!
>>
>> I am soliciting feedbacks for HDFS roadmap items and wish list in the
>> future Hadoop releases. A community meetup
>> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
>> is happening soon, and perhaps we can use this thread to converge on things
>> we should talk about there.
>>
>> I am aware of several major features that merged into trunk, such as RBF,
>> Consistent Standby Serving Reads, as well as some recent features that
>> merged into 3.2.0 release (storage policy satisfier).
>>
>> What else should we be doing? I have a laundry list of supportability
>> improvement projects, mostly about improving performance or making
>> performance diagnostics easier. I can share the list if folks are
>> interested.
>>
>> Are there things we should do to make developer's life easier or things
>> that would be nice to have for downstream applications? I know Sahil Takiar
>> made a series of improvements in HDFS for Impala recently, and those
>> improvements are applicable to other downstreamers such as HBase. Or would
>> it help if we provide more Hadoop API examples?
>>
>

Re: [DISCUSS] HDFS roadmap/wish list

Posted by Julien Laurenceau <ju...@pepitedata.com>.
Hi,

I am not absolutely sure it is not already in a roadmap or supported, but I
would appreciate those two features :

- First feature : I would also like to be able to use a dedicated directory
in HDFS as a /tmp directory leveraging RAMFS for high performing checkpoint
of Spark Jobs without using Alluxio or Ignite.
My current issue is that the RAMFS is only useful with replication factor
x1 (in order to avoid network).
My default replication factor is x3, but I would need a way to set
replication factor x1 on a specific directory (/tmp) for all new writes
coming to this directory.
Currently if I use "hdfs setrep 1 /tmp" it only works for blocks already
written.
For example, this could be done by specifying the replication factor at the
storage policy level.
In my view this would dramatically improve the interest of the Lazy-persist
storage policy.

> From the Doc > Note 1: The Lazy_Persist policy is useful only for single
replica blocks. For blocks with more than one replicas, all the replicas
will be written to DISK since writing only one of the replicas to RAM_DISK
does not improve the overall performance.
In the current state of HDFS configuration, I only see the following hack
(not tested) to implement such a solution : Configure HDFS replication x1
as default configuration and use Erasure Coding RS(6,3) for the main
storage by attaching an ec storage policy on all directories except /tmp.

hdfs ec -setPolicy -path <directory> [-policy <policyName>]



- Second feature: a bandwidth throttling dedicated to the re-replication in
case of a failed datanode.
Something similar to the option dedicated to the balancing algorithm
dfs.datanode.balance.bandwidthPerSecbut only for re-replication.

Thanks and regards
JL

Le lun. 10 juin 2019 à 19:08, Wei-Chiu Chuang <we...@cloudera.com.invalid>
a écrit :

> Hi!
>
> I am soliciting feedbacks for HDFS roadmap items and wish list in the
> future Hadoop releases. A community meetup
> <https://www.meetup.com/Hadoop-Contributors/events/262055924/?rv=ea1_v2&_xtd=gatlbWFpbF9jbGlja9oAJGJiNTE1ODdkLTY0MDAtNDFiZS1iOTU5LTM5ZWYyMDU1N2Q4Nw>
> is happening soon, and perhaps we can use this thread to converge on things
> we should talk about there.
>
> I am aware of several major features that merged into trunk, such as RBF,
> Consistent Standby Serving Reads, as well as some recent features that
> merged into 3.2.0 release (storage policy satisfier).
>
> What else should we be doing? I have a laundry list of supportability
> improvement projects, mostly about improving performance or making
> performance diagnostics easier. I can share the list if folks are
> interested.
>
> Are there things we should do to make developer's life easier or things
> that would be nice to have for downstream applications? I know Sahil Takiar
> made a series of improvements in HDFS for Impala recently, and those
> improvements are applicable to other downstreamers such as HBase. Or would
> it help if we provide more Hadoop API examples?
>