You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hudi.apache.org by "Scheller, Brandon" <bs...@amazon.com.INVALID> on 2019/11/08 22:48:44 UTC
[Discuss] Feedback on Hudi improvements
Hi Hudi community,
We at AWS EMR are interested in starting work on a few different usability improvements for Hudi and we’re interested to hear your feedback.
Here are some of our ideas:
https://issues.apache.org/jira/browse/HUDI-326
https://issues.apache.org/jira/browse/HUDI-327
Additionally, we were hoping to help drive:
https://issues.apache.org/jira/browse/HUDI-83 and its associated Hive Jira: https://issues.apache.org/jira/browse/HIVE-22224
I am looking forward to improving Hudi with you all. And feel free to let us know if there is anything specific, you’d like us to look at.
Thanks,
Brandon
Re: [Discuss] Feedback on Hudi improvements
Posted by Vinoth Chandar <vi...@apache.org>.
Please go ahead. If you can share your jira ID, we can add you to
contributors to get the ticket assigned to you
On Mon, Nov 11, 2019 at 1:14 PM Scheller, Brandon
<bs...@amazon.com.invalid> wrote:
> Yep, you are correct that it is throwing the exception because of the
> DataSourceUtils.getNestedFieldValAsString.
> I can take up the work to fix this behavior if it is not intended. I'd
> also like to add extra error messaging and validation because currently it
> is not clear to users what the error is when a record_key is empty. It just
> throws the following and requires trace debugging to root cause:
> 19/11/11 21:09:36 ERROR HoodieSparkSqlWriter$: insert failed with 1 errors
> :
>
> -Brandon
>
> On 11/10/19, 11:24 PM, "Jaimin Shah" <sh...@gmail.com> wrote:
>
> Hi Brandon,
> I contributed to complex ComplexKeyGenerator sometime back. I don't
> think
> it is intended behavior. If you are getting exception is it because of
> DataSourceUtils.getNestedFieldValAsString(record, recordKeyField) ? I
> can't
> think of any other reason why it should throw exception.
> I think when val is null DataSourceUtils.getNestedFieldValAsString
> throw
> error due to
> if (!(val instanceof GenericRecord)) {
> throw new HoodieException("Cannot find a record at part value :" +
> part);
> }
> Maybe Balaji can confirm it.
>
> Thanks,
> Jaimin
>
> On Sun, 10 Nov 2019 at 03:40, Scheller, Brandon
> <bs...@amazon.com.invalid>
> wrote:
>
> > Thanks for the quick response Balaji!
> >
> > I think there is a lot here to continue with:
> > 1. I did see that recent pull request for the delete API. I think
> > collaborating to support another delete API with just record key
> would be a
> > great next step. I'll begin looking into it. Additionally, the
> scenario of
> > using Hbase as the global index is definitely something which we'd be
> > interested in understanding further.
> > 2. Actually I was speaking to the case of ComplexKeyGenerator.
> Currently
> > if any single component of it is null, it will throw an exception.
> If this
> > is not intended behavior, I'd be happy to fix this bug as it looks
> to solve
> > our use case.
> > 3. Thanks for the update on this. The spark upgrade is definitely a
> large
> > undertaking that I'd be happy to help with.
> >
> > Thanks again,
> > Brandon
> >
> > On 11/8/19, 3:52 PM, "Balaji Varadarajan" <va...@gmail.com>
> wrote:
> >
> > Brandon,
> >
> > Great initiative and thoughts. Thanks for writing detailed
> description
> > on
> > what you are looking to achieve.
> >
> > Here are some of my comments/thoughts:
> >
> > 1. HUDI-326 : There is some work that is happening in this
> > direction.
> > But, we should be able to collaborate on this. Siva has
> opened a PR
> > (
> > https://github.com/apache/incubator-hudi/pull/1004) to
> support
> > delete
> > using only HoodieKey (partitionPath, recordKey). Technically,
> we can
> > support an interface for delete with only recordKeys if the
> index
> > is of
> > type global (Current implementation supports
> > HoodieGlobalBloomIndex).
> > Within Uber, we use Hbase as the global Hudi index to support
> > partition
> > agnostic record-key lookups. In other words, we can have 2
> flavors
> > of
> > delete APIs - one with input being RDD<HoodieKeys> (works for
> all
> > index
> > types) and another with input RDD<RecordKey> that works with
> global
> > index. Our vision is to support an external clustered index
> > (global) as the
> > de-facto index that resides in DFS along with dataset.
> > 2. HUDI-327 : IIUC, Just like ComplexKeyGenerator, the new
> key
> > generator would need composite keys (in this case primary and
> > secondary for
> > breaking the "null" tie ). Are you concerned about the
> record-key
> > footprint
> > for each key when using the key generated by
> ComplexKeyGenerator?
> > In that
> > case, makes sense to me. Otherwise, ComplexKeyGenerator
> should be
> > able to
> > handle cases when some component of it is null. right ?
> > 3. As for HUDI-83, at-least on the write side, we have tied
> this
> > with
> > spark-2.4 upgrade. There is ongoing work happening in this
> regard.
> > I will
> > request folks who is working on this to provide status. Last I
> > know, we
> > were running into some test failures when doing this
> upgrade. But
> > yes, as
> > this is a massive upgrade, we would need your help in
> reviewing,
> > debugging
> > and testing this change :)
> >
> > Others, Thoughts ?
> >
> > Thanks,
> > Balaji.V
> >
> > On Fri, Nov 8, 2019 at 2:49 PM Scheller, Brandon
> > <bs...@amazon.com.invalid> wrote:
> >
> > > Hi Hudi community,
> > >
> > > We at AWS EMR are interested in starting work on a few
> different
> > usability
> > > improvements for Hudi and we’re interested to hear your
> feedback.
> > >
> > > Here are some of our ideas:
> > > https://issues.apache.org/jira/browse/HUDI-326
> > > https://issues.apache.org/jira/browse/HUDI-327
> > >
> > > Additionally, we were hoping to help drive:
> > > https://issues.apache.org/jira/browse/HUDI-83 and its
> associated
> > Hive
> > > Jira: https://issues.apache.org/jira/browse/HIVE-22224
> > >
> > > I am looking forward to improving Hudi with you all. And feel
> free
> > to let
> > > us know if there is anything specific, you’d like us to look
> at.
> > >
> > > Thanks,
> > > Brandon
> > >
> >
> >
> >
>
>
>
Re: [Discuss] Feedback on Hudi improvements
Posted by "Scheller, Brandon" <bs...@amazon.com.INVALID>.
Yep, you are correct that it is throwing the exception because of the DataSourceUtils.getNestedFieldValAsString.
I can take up the work to fix this behavior if it is not intended. I'd also like to add extra error messaging and validation because currently it is not clear to users what the error is when a record_key is empty. It just throws the following and requires trace debugging to root cause:
19/11/11 21:09:36 ERROR HoodieSparkSqlWriter$: insert failed with 1 errors :
-Brandon
On 11/10/19, 11:24 PM, "Jaimin Shah" <sh...@gmail.com> wrote:
Hi Brandon,
I contributed to complex ComplexKeyGenerator sometime back. I don't think
it is intended behavior. If you are getting exception is it because of
DataSourceUtils.getNestedFieldValAsString(record, recordKeyField) ? I can't
think of any other reason why it should throw exception.
I think when val is null DataSourceUtils.getNestedFieldValAsString throw
error due to
if (!(val instanceof GenericRecord)) {
throw new HoodieException("Cannot find a record at part value :" + part);
}
Maybe Balaji can confirm it.
Thanks,
Jaimin
On Sun, 10 Nov 2019 at 03:40, Scheller, Brandon <bs...@amazon.com.invalid>
wrote:
> Thanks for the quick response Balaji!
>
> I think there is a lot here to continue with:
> 1. I did see that recent pull request for the delete API. I think
> collaborating to support another delete API with just record key would be a
> great next step. I'll begin looking into it. Additionally, the scenario of
> using Hbase as the global index is definitely something which we'd be
> interested in understanding further.
> 2. Actually I was speaking to the case of ComplexKeyGenerator. Currently
> if any single component of it is null, it will throw an exception. If this
> is not intended behavior, I'd be happy to fix this bug as it looks to solve
> our use case.
> 3. Thanks for the update on this. The spark upgrade is definitely a large
> undertaking that I'd be happy to help with.
>
> Thanks again,
> Brandon
>
> On 11/8/19, 3:52 PM, "Balaji Varadarajan" <va...@gmail.com> wrote:
>
> Brandon,
>
> Great initiative and thoughts. Thanks for writing detailed description
> on
> what you are looking to achieve.
>
> Here are some of my comments/thoughts:
>
> 1. HUDI-326 : There is some work that is happening in this
> direction.
> But, we should be able to collaborate on this. Siva has opened a PR
> (
> https://github.com/apache/incubator-hudi/pull/1004) to support
> delete
> using only HoodieKey (partitionPath, recordKey). Technically, we can
> support an interface for delete with only recordKeys if the index
> is of
> type global (Current implementation supports
> HoodieGlobalBloomIndex).
> Within Uber, we use Hbase as the global Hudi index to support
> partition
> agnostic record-key lookups. In other words, we can have 2 flavors
> of
> delete APIs - one with input being RDD<HoodieKeys> (works for all
> index
> types) and another with input RDD<RecordKey> that works with global
> index. Our vision is to support an external clustered index
> (global) as the
> de-facto index that resides in DFS along with dataset.
> 2. HUDI-327 : IIUC, Just like ComplexKeyGenerator, the new key
> generator would need composite keys (in this case primary and
> secondary for
> breaking the "null" tie ). Are you concerned about the record-key
> footprint
> for each key when using the key generated by ComplexKeyGenerator?
> In that
> case, makes sense to me. Otherwise, ComplexKeyGenerator should be
> able to
> handle cases when some component of it is null. right ?
> 3. As for HUDI-83, at-least on the write side, we have tied this
> with
> spark-2.4 upgrade. There is ongoing work happening in this regard.
> I will
> request folks who is working on this to provide status. Last I
> know, we
> were running into some test failures when doing this upgrade. But
> yes, as
> this is a massive upgrade, we would need your help in reviewing,
> debugging
> and testing this change :)
>
> Others, Thoughts ?
>
> Thanks,
> Balaji.V
>
> On Fri, Nov 8, 2019 at 2:49 PM Scheller, Brandon
> <bs...@amazon.com.invalid> wrote:
>
> > Hi Hudi community,
> >
> > We at AWS EMR are interested in starting work on a few different
> usability
> > improvements for Hudi and we’re interested to hear your feedback.
> >
> > Here are some of our ideas:
> > https://issues.apache.org/jira/browse/HUDI-326
> > https://issues.apache.org/jira/browse/HUDI-327
> >
> > Additionally, we were hoping to help drive:
> > https://issues.apache.org/jira/browse/HUDI-83 and its associated
> Hive
> > Jira: https://issues.apache.org/jira/browse/HIVE-22224
> >
> > I am looking forward to improving Hudi with you all. And feel free
> to let
> > us know if there is anything specific, you’d like us to look at.
> >
> > Thanks,
> > Brandon
> >
>
>
>
Re: [Discuss] Feedback on Hudi improvements
Posted by Jaimin Shah <sh...@gmail.com>.
Hi Brandon,
I contributed to complex ComplexKeyGenerator sometime back. I don't think
it is intended behavior. If you are getting exception is it because of
DataSourceUtils.getNestedFieldValAsString(record, recordKeyField) ? I can't
think of any other reason why it should throw exception.
I think when val is null DataSourceUtils.getNestedFieldValAsString throw
error due to
if (!(val instanceof GenericRecord)) {
throw new HoodieException("Cannot find a record at part value :" + part);
}
Maybe Balaji can confirm it.
Thanks,
Jaimin
On Sun, 10 Nov 2019 at 03:40, Scheller, Brandon <bs...@amazon.com.invalid>
wrote:
> Thanks for the quick response Balaji!
>
> I think there is a lot here to continue with:
> 1. I did see that recent pull request for the delete API. I think
> collaborating to support another delete API with just record key would be a
> great next step. I'll begin looking into it. Additionally, the scenario of
> using Hbase as the global index is definitely something which we'd be
> interested in understanding further.
> 2. Actually I was speaking to the case of ComplexKeyGenerator. Currently
> if any single component of it is null, it will throw an exception. If this
> is not intended behavior, I'd be happy to fix this bug as it looks to solve
> our use case.
> 3. Thanks for the update on this. The spark upgrade is definitely a large
> undertaking that I'd be happy to help with.
>
> Thanks again,
> Brandon
>
> On 11/8/19, 3:52 PM, "Balaji Varadarajan" <va...@gmail.com> wrote:
>
> Brandon,
>
> Great initiative and thoughts. Thanks for writing detailed description
> on
> what you are looking to achieve.
>
> Here are some of my comments/thoughts:
>
> 1. HUDI-326 : There is some work that is happening in this
> direction.
> But, we should be able to collaborate on this. Siva has opened a PR
> (
> https://github.com/apache/incubator-hudi/pull/1004) to support
> delete
> using only HoodieKey (partitionPath, recordKey). Technically, we can
> support an interface for delete with only recordKeys if the index
> is of
> type global (Current implementation supports
> HoodieGlobalBloomIndex).
> Within Uber, we use Hbase as the global Hudi index to support
> partition
> agnostic record-key lookups. In other words, we can have 2 flavors
> of
> delete APIs - one with input being RDD<HoodieKeys> (works for all
> index
> types) and another with input RDD<RecordKey> that works with global
> index. Our vision is to support an external clustered index
> (global) as the
> de-facto index that resides in DFS along with dataset.
> 2. HUDI-327 : IIUC, Just like ComplexKeyGenerator, the new key
> generator would need composite keys (in this case primary and
> secondary for
> breaking the "null" tie ). Are you concerned about the record-key
> footprint
> for each key when using the key generated by ComplexKeyGenerator?
> In that
> case, makes sense to me. Otherwise, ComplexKeyGenerator should be
> able to
> handle cases when some component of it is null. right ?
> 3. As for HUDI-83, at-least on the write side, we have tied this
> with
> spark-2.4 upgrade. There is ongoing work happening in this regard.
> I will
> request folks who is working on this to provide status. Last I
> know, we
> were running into some test failures when doing this upgrade. But
> yes, as
> this is a massive upgrade, we would need your help in reviewing,
> debugging
> and testing this change :)
>
> Others, Thoughts ?
>
> Thanks,
> Balaji.V
>
> On Fri, Nov 8, 2019 at 2:49 PM Scheller, Brandon
> <bs...@amazon.com.invalid> wrote:
>
> > Hi Hudi community,
> >
> > We at AWS EMR are interested in starting work on a few different
> usability
> > improvements for Hudi and we’re interested to hear your feedback.
> >
> > Here are some of our ideas:
> > https://issues.apache.org/jira/browse/HUDI-326
> > https://issues.apache.org/jira/browse/HUDI-327
> >
> > Additionally, we were hoping to help drive:
> > https://issues.apache.org/jira/browse/HUDI-83 and its associated
> Hive
> > Jira: https://issues.apache.org/jira/browse/HIVE-22224
> >
> > I am looking forward to improving Hudi with you all. And feel free
> to let
> > us know if there is anything specific, you’d like us to look at.
> >
> > Thanks,
> > Brandon
> >
>
>
>
Re: [Discuss] Feedback on Hudi improvements
Posted by "Scheller, Brandon" <bs...@amazon.com.INVALID>.
Thanks for the quick response Balaji!
I think there is a lot here to continue with:
1. I did see that recent pull request for the delete API. I think collaborating to support another delete API with just record key would be a great next step. I'll begin looking into it. Additionally, the scenario of using Hbase as the global index is definitely something which we'd be interested in understanding further.
2. Actually I was speaking to the case of ComplexKeyGenerator. Currently if any single component of it is null, it will throw an exception. If this is not intended behavior, I'd be happy to fix this bug as it looks to solve our use case.
3. Thanks for the update on this. The spark upgrade is definitely a large undertaking that I'd be happy to help with.
Thanks again,
Brandon
On 11/8/19, 3:52 PM, "Balaji Varadarajan" <va...@gmail.com> wrote:
Brandon,
Great initiative and thoughts. Thanks for writing detailed description on
what you are looking to achieve.
Here are some of my comments/thoughts:
1. HUDI-326 : There is some work that is happening in this direction.
But, we should be able to collaborate on this. Siva has opened a PR (
https://github.com/apache/incubator-hudi/pull/1004) to support delete
using only HoodieKey (partitionPath, recordKey). Technically, we can
support an interface for delete with only recordKeys if the index is of
type global (Current implementation supports HoodieGlobalBloomIndex).
Within Uber, we use Hbase as the global Hudi index to support partition
agnostic record-key lookups. In other words, we can have 2 flavors of
delete APIs - one with input being RDD<HoodieKeys> (works for all index
types) and another with input RDD<RecordKey> that works with global
index. Our vision is to support an external clustered index (global) as the
de-facto index that resides in DFS along with dataset.
2. HUDI-327 : IIUC, Just like ComplexKeyGenerator, the new key
generator would need composite keys (in this case primary and secondary for
breaking the "null" tie ). Are you concerned about the record-key footprint
for each key when using the key generated by ComplexKeyGenerator? In that
case, makes sense to me. Otherwise, ComplexKeyGenerator should be able to
handle cases when some component of it is null. right ?
3. As for HUDI-83, at-least on the write side, we have tied this with
spark-2.4 upgrade. There is ongoing work happening in this regard. I will
request folks who is working on this to provide status. Last I know, we
were running into some test failures when doing this upgrade. But yes, as
this is a massive upgrade, we would need your help in reviewing, debugging
and testing this change :)
Others, Thoughts ?
Thanks,
Balaji.V
On Fri, Nov 8, 2019 at 2:49 PM Scheller, Brandon
<bs...@amazon.com.invalid> wrote:
> Hi Hudi community,
>
> We at AWS EMR are interested in starting work on a few different usability
> improvements for Hudi and we’re interested to hear your feedback.
>
> Here are some of our ideas:
> https://issues.apache.org/jira/browse/HUDI-326
> https://issues.apache.org/jira/browse/HUDI-327
>
> Additionally, we were hoping to help drive:
> https://issues.apache.org/jira/browse/HUDI-83 and its associated Hive
> Jira: https://issues.apache.org/jira/browse/HIVE-22224
>
> I am looking forward to improving Hudi with you all. And feel free to let
> us know if there is anything specific, you’d like us to look at.
>
> Thanks,
> Brandon
>
Re: [Discuss] Feedback on Hudi improvements
Posted by Balaji Varadarajan <va...@gmail.com>.
Brandon,
Great initiative and thoughts. Thanks for writing detailed description on
what you are looking to achieve.
Here are some of my comments/thoughts:
1. HUDI-326 : There is some work that is happening in this direction.
But, we should be able to collaborate on this. Siva has opened a PR (
https://github.com/apache/incubator-hudi/pull/1004) to support delete
using only HoodieKey (partitionPath, recordKey). Technically, we can
support an interface for delete with only recordKeys if the index is of
type global (Current implementation supports HoodieGlobalBloomIndex).
Within Uber, we use Hbase as the global Hudi index to support partition
agnostic record-key lookups. In other words, we can have 2 flavors of
delete APIs - one with input being RDD<HoodieKeys> (works for all index
types) and another with input RDD<RecordKey> that works with global
index. Our vision is to support an external clustered index (global) as the
de-facto index that resides in DFS along with dataset.
2. HUDI-327 : IIUC, Just like ComplexKeyGenerator, the new key
generator would need composite keys (in this case primary and secondary for
breaking the "null" tie ). Are you concerned about the record-key footprint
for each key when using the key generated by ComplexKeyGenerator? In that
case, makes sense to me. Otherwise, ComplexKeyGenerator should be able to
handle cases when some component of it is null. right ?
3. As for HUDI-83, at-least on the write side, we have tied this with
spark-2.4 upgrade. There is ongoing work happening in this regard. I will
request folks who is working on this to provide status. Last I know, we
were running into some test failures when doing this upgrade. But yes, as
this is a massive upgrade, we would need your help in reviewing, debugging
and testing this change :)
Others, Thoughts ?
Thanks,
Balaji.V
On Fri, Nov 8, 2019 at 2:49 PM Scheller, Brandon
<bs...@amazon.com.invalid> wrote:
> Hi Hudi community,
>
> We at AWS EMR are interested in starting work on a few different usability
> improvements for Hudi and we’re interested to hear your feedback.
>
> Here are some of our ideas:
> https://issues.apache.org/jira/browse/HUDI-326
> https://issues.apache.org/jira/browse/HUDI-327
>
> Additionally, we were hoping to help drive:
> https://issues.apache.org/jira/browse/HUDI-83 and its associated Hive
> Jira: https://issues.apache.org/jira/browse/HIVE-22224
>
> I am looking forward to improving Hudi with you all. And feel free to let
> us know if there is anything specific, you’d like us to look at.
>
> Thanks,
> Brandon
>