[DISCUSS] Simplification of terminologies

[Vinoth Chandar <vi...@apache.org>]

Hello all,

I wanted to raise an important topic with the community around whether we
should rename some of our terminologies in code/docs to be more
user-friendly and understandable..

Let me also provide some context for each, since I am probably guilty of
introducing most of them in the first place :).

*1. Rename "views" to "query" : *Instead of saying incremental view or
read-optimized view, talk about them as "incremental query" and
"read-optimized query". The term "view" is very technical, and what I was
trying to convey was that we ingest/store the data once and expose views on
top. But new users (atleast half dozen of them to me) tend to confuse this
with views/materialized views found in databases. Almost always we talk
about views mostly in terms of expected behavior for a query on the view. I
am proposing to just call these different query types since its a more
universally accepted terminology and IMO clearer.

*2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
Read-Optimized view only for MOR storage :* This one is probably the
trickiest. Hudi was always designed with MOR in mind, even as we were
working on COW storage and consequently we named the pure parquet backed
view as Read-Optimized, hoping to name parquet + avro based view as
Write-Optimized. However, we opted to name it Realtime to emphasize the
data freshness aspect. In retrospect, the views should have not been named
after their performance characteristics but rather the classes of queries
done on them and guarantees for those (point above #1). Moreover, once we
have parquet embedded into the log format, then the tradeoffs may not be
the same anyways.

So combining with the renaming proposed in #1, we would end up with the
following..

Copy-On-Write :
[Old]  Read-Optimized View =>  [New] Snapshot Query
[Old]  Incremental View => [New] Incremental Query

Merge-On-Read:
[Old] Realtime View => [New] Snapshot Query
[Old] Incremental View => [New] Incremental Query
[Old] ReadOptimzied View => [New] Read-Optimized Query (since it is read
optimized compared to Snapshot query always, at the cost of staler data)

Both changes #1 & #2 could be simpler changes to just code references, docs
and configs.. we can support both string for sometime and deprecate
eventually since queries are stateless.

*3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since the
design was very similar to https://en.wikipedia.org/wiki/Copy-on-write
filesystems
& snapshotting and we once hoped to push some of this logic into the
storage itself, all in vain. but the name stuck, even though once we had
MERGE_ON_READ the focus was often on merge costs etc, which the name
COPY_ON_WRITE does not convey directly. I don't feel very strong about this
and there is also cost to changing this since its persisted inside
hoodie.properties and we will support both strings internally in code for
backwards compatibility anyway

Naming something is very hard (yes, try :)).I believe these changes will
make the project simpler to understand for everyone out there. We also have
tons of new people here, so I am also happy to let go, if its already clear
:)

Please use the bullet number when you share your feedback so we know what
the discussion is about.

Thanks
Vinoth

Re: [DISCUSS] Simplification of terminologies

[Pratyaksh Sharma <pr...@gmail.com>]

All the three proposals seem to make life easier for Hudi users. +1 for all
of them.

On Wed, Nov 13, 2019 at 8:34 AM Vinoth Chandar <vi...@apache.org> wrote:

> Thanks everyone for the feedback. Looks like we are in general agreement.
>
> I am inclined to just do 1 & 2 and leave COPY_ON_WRITE as is based on great
> points Ethan and Shiyan raised. Makes sense..
> Will wait for 1-2 days still to close this thread.
>
> @semanticbeeing Thats a great idea. Is it more like a technical glossary of
> sorts? Lets may be start a different DISCUSS thread on that specific topic,
> so everyone can chime in and provide more attention to that proposal?
>
>
>
>
>
> On Tue, Nov 12, 2019 at 2:44 PM Y. Ethan Guo <gu...@uber.com.invalid>
> wrote:
>
> > +1 on [1] and [2].
> >
> > For [3], I have similar doubts as Shiyan.
> >
> > For the naming, I can understand the original intent of the analogy for
> COW
> > which is to make another "copy" of columnar/parquet file upon the
> > modification/update to the records in the file.  From the system design
> > point of view, it's easy to understand.  I'm okay with the renaming as
> > "MERGE_ON_WRITE" since it's probably straightforward for users at the
> first
> > glance.
> >
> > In terms of the concept, COW and MOR are listed as storage/table types.
> > From my understanding, they represent different tradeoffs of the
> > performance between reading and writing Hudi tables, and within MOR there
> > are different tradeoffs, e.g., lazy merge on read or periodic compaction
> > and cleaning pipelined along ingestion. It looks like these can be
> > controlled through configs, e.g., "disable_merge_on_write",
> > "compaction_frenquency", etc., instead of fixing the storage type, to
> > control the tradeoff that a user would like to make.  The requirement may
> > change so a user can switch between COW and MOR by tuning the configs. We
> > don't have to make such changes now, but I'm wondering if this is
> something
> > worth considering in the future releases.
> >
> > - Ethan
> >
> > On Tue, Nov 12, 2019 at 8:43 AM nishith agarwal <n3...@gmail.com>
> > wrote:
> >
> > > +1 on the first two, don't feel strongly about (3).
> > >
> > > Thanks,
> > > Nishith
> > >
> > > On Tue, Nov 12, 2019 at 5:03 AM leesf <le...@gmail.com> wrote:
> > >
> > > > [1] +1. `views` indeed confused me a lot.
> > > > [2] +1. `snapshot` is more reasonable.
> > > > [3] I don't feel very strong to rename it, the current name
> > > `COPY_ON_WRITE`
> > > > is reasonable considering the cost to rename and the behavior that
> new
> > > > version parquet file will be created and seems to be copied from old
> > > > version parquet file.
> > > >
> > > > Best,
> > > > Leesf
> > > >
> > > > Balaji Varadarajan <vb...@apache.org> 于2019年11月12日周二 下午3:55写道：
> > > >
> > > > > Agree with all 3 changes. The naming now looks more consistent than
> > > > > earlier. +1 on them
> > > > >
> > > > > Depending on whether we are renaming Input formats for (1) and (2)
> -
> > > this
> > > > > could require some migration steps for
> > > > >
> > > > > Balaji.V
> > > > >
> > > > >
> > > > > On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hi Vinoth,
> > > > > >
> > > > > > Thanks for bringing these proposals.
> > > > > >
> > > > > > +1 on all three. Especially, big +1 on the third renaming
> proposal.
> > > > > >
> > > > > > When I was a newbie. The "COPY_ON_WRITE" term confused me a lot.
> It
> > > > > easily
> > > > > > mislead users on the "copy" term. And make users compare it with
> > the
> > > > > > `CopyOnWriteArrayList` data structure provided by JDK  and
> thoughts
> > > of
> > > > > the
> > > > > > file systems.
> > > > > >
> > > > > > Best,
> > > > > > Vino
> > > > > >
> > > > > >
> > > > > > Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
> > > > > >
> > > > > > > +1 on all three rename proposals. I think this would make the
> > > > concepts
> > > > > > > super easy to follow for new users.
> > > > > > >
> > > > > > > If changing [3] seems to be a stretch, we should definitely do
> > [1]
> > > &
> > > > > [2]
> > > > > > at
> > > > > > > the least IMO. I will be glad to help out on the renames to
> > > whatever
> > > > > > extent
> > > > > > > possible should the Hudi community incline to pursue this.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sudha
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <
> > vinoth@apache.org>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hello all,
> > > > > > > >
> > > > > > > > I wanted to raise an important topic with the community
> around
> > > > > whether
> > > > > > we
> > > > > > > > should rename some of our terminologies in code/docs to be
> more
> > > > > > > > user-friendly and understandable..
> > > > > > > >
> > > > > > > > Let me also provide some context for each, since I am
> probably
> > > > guilty
> > > > > > of
> > > > > > > > introducing most of them in the first place :).
> > > > > > > >
> > > > > > > > *1. Rename "views" to "query" : *Instead of saying
> incremental
> > > view
> > > > > or
> > > > > > > > read-optimized view, talk about them as "incremental query"
> and
> > > > > > > > "read-optimized query". The term "view" is very technical,
> and
> > > > what I
> > > > > > was
> > > > > > > > trying to convey was that we ingest/store the data once and
> > > expose
> > > > > > views
> > > > > > > on
> > > > > > > > top. But new users (atleast half dozen of them to me) tend to
> > > > confuse
> > > > > > > this
> > > > > > > > with views/materialized views found in databases. Almost
> always
> > > we
> > > > > talk
> > > > > > > > about views mostly in terms of expected behavior for a query
> on
> > > the
> > > > > > > view. I
> > > > > > > > am proposing to just call these different query types since
> > its a
> > > > > more
> > > > > > > > universally accepted terminology and IMO clearer.
> > > > > > > >
> > > > > > > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views
> +
> > > Have
> > > > > > > > Read-Optimized view only for MOR storage :* This one is
> > probably
> > > > the
> > > > > > > > trickiest. Hudi was always designed with MOR in mind, even as
> > we
> > > > were
> > > > > > > > working on COW storage and consequently we named the pure
> > parquet
> > > > > > backed
> > > > > > > > view as Read-Optimized, hoping to name parquet + avro based
> > view
> > > as
> > > > > > > > Write-Optimized. However, we opted to name it Realtime to
> > > emphasize
> > > > > the
> > > > > > > > data freshness aspect. In retrospect, the views should have
> not
> > > > been
> > > > > > > named
> > > > > > > > after their performance characteristics but rather the
> classes
> > of
> > > > > > queries
> > > > > > > > done on them and guarantees for those (point above #1).
> > Moreover,
> > > > > once
> > > > > > we
> > > > > > > > have parquet embedded into the log format, then the tradeoffs
> > may
> > > > not
> > > > > > be
> > > > > > > > the same anyways.
> > > > > > > >
> > > > > > > > So combining with the renaming proposed in #1, we would end
> up
> > > with
> > > > > the
> > > > > > > > following..
> > > > > > > >
> > > > > > > > Copy-On-Write :
> > > > > > > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > > > > > > [Old]  Incremental View => [New] Incremental Query
> > > > > > > >
> > > > > > > > Merge-On-Read:
> > > > > > > > [Old] Realtime View => [New] Snapshot Query
> > > > > > > > [Old] Incremental View => [New] Incremental Query
> > > > > > > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since
> > it
> > > is
> > > > > > read
> > > > > > > > optimized compared to Snapshot query always, at the cost of
> > > staler
> > > > > > data)
> > > > > > > >
> > > > > > > > Both changes #1 & #2 could be simpler changes to just code
> > > > > references,
> > > > > > > docs
> > > > > > > > and configs.. we can support both string for sometime and
> > > deprecate
> > > > > > > > eventually since queries are stateless.
> > > > > > > >
> > > > > > > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated
> > > since
> > > > > the
> > > > > > > > design was very similar to
> > > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Copy-2Don-2Dwrite&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=y9XF8-75xzGHY4yCbfVVWcIC1sbEXDxitqeAS2A6GoQ&e=
> > > > > > > > filesystems
> > > > > > > > & snapshotting and we once hoped to push some of this logic
> > into
> > > > the
> > > > > > > > storage itself, all in vain. but the name stuck, even though
> > once
> > > > we
> > > > > > had
> > > > > > > > MERGE_ON_READ the focus was often on merge costs etc, which
> the
> > > > name
> > > > > > > > COPY_ON_WRITE does not convey directly. I don't feel very
> > strong
> > > > > about
> > > > > > > this
> > > > > > > > and there is also cost to changing this since its persisted
> > > inside
> > > > > > > >
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__hoodie.properties&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=930ugGMXsrqzE-acg9nfeoePBmVjTRG3gD765ihEiqU&e=
> > > and we will support both strings internally in
> > > > code
> > > > > > for
> > > > > > > > backwards compatibility anyway
> > > > > > > >
> > > > > > > > Naming something is very hard (yes, try :)).I believe these
> > > changes
> > > > > > will
> > > > > > > > make the project simpler to understand for everyone out
> there.
> > We
> > > > > also
> > > > > > > have
> > > > > > > > tons of new people here, so I am also happy to let go, if its
> > > > already
> > > > > > > clear
> > > > > > > > :)
> > > > > > > >
> > > > > > > > Please use the bullet number when you share your feedback so
> we
> > > > know
> > > > > > what
> > > > > > > > the discussion is about.
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Vinoth
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

[Vinoth Chandar <vi...@apache.org>]

Howdy all,

I have written up a first full version of
https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture
<https://cwiki.apache.org/confluence/display/HUDI/Design+And+Architecture>,
based on style set by Nick on the cWiki.
Please contribute to making this more elaborate.

Special thanks to Nick, for introducing a great way to fully leverage power
of wiki and the patience in working together on seeding this.

Moving onto cleaning up the code classes and docs.

On Wed, Nov 13, 2019 at 9:24 PM Vinoth Chandar <vi...@apache.org> wrote:

> Will review the POC in cwiki.  +1
>
> Based on this feedback, I will proceed with the changes. Thanks all!
>
>
>
> On Tue, Nov 12, 2019 at 10:47 PM Semantic Beeng <ni...@semanticbeeng.com>
> wrote:
>
>> @vc, I think of it as elaborating the #ubiquitouslanguage in DDD.
>> See private email with references to a small POC in wiki and decide how
>> to proceed.
>>
>> On November 12, 2019 at 10:04 PM Vinoth Chandar < vinoth@apache.org>
>> wrote:
>>
>>
>> Thanks everyone for the feedback. Looks like we are in general agreement.
>>
>> I am inclined to just do 1 & 2 and leave COPY_ON_WRITE as is based on
>> great
>> points Ethan and Shiyan raised. Makes sense..
>> Will wait for 1-2 days still to close this thread.
>>
>> @semanticbeeing Thats a great idea. Is it more like a technical glossary
>> of
>> sorts? Lets may be start a different DISCUSS thread on that specific
>> topic,
>> so everyone can chime in and provide more attention to that proposal?
>>
>>
>>
>>
>>
>> On Tue, Nov 12, 2019 at 2:44 PM Y. Ethan Guo < guoyihua@uber.com.invalid>
>>
>> wrote:
>>
>> +1 on [1] and [2].
>>
>> For [3], I have similar doubts as Shiyan.
>>
>> For the naming, I can understand the original intent of the analogy for
>> COW
>> which is to make another "copy" of columnar/parquet file upon the
>> modification/update to the records in the file. From the system design
>> point of view, it's easy to understand. I'm okay with the renaming as
>> "MERGE_ON_WRITE" since it's probably straightforward for users at the
>> first
>> glance.
>>
>> In terms of the concept, COW and MOR are listed as storage/table types.
>> From my understanding, they represent different tradeoffs of the
>> performance between reading and writing Hudi tables, and within MOR there
>> are different tradeoffs, e.g., lazy merge on read or periodic compaction
>> and cleaning pipelined along ingestion. It looks like these can be
>> controlled through configs, e.g., "disable_merge_on_write",
>> "compaction_frenquency", etc., instead of fixing the storage type, to
>> control the tradeoff that a user would like to make. The requirement may
>> change so a user can switch between COW and MOR by tuning the configs. We
>> don't have to make such changes now, but I'm wondering if this is
>> something
>> worth considering in the future releases.
>>
>> - Ethan
>>
>> On Tue, Nov 12, 2019 at 8:43 AM nishith agarwal < n3.nash29@gmail.com>
>> wrote:
>>
>> +1 on the first two, don't feel strongly about (3).
>>
>> Thanks,
>> Nishith
>>
>> On Tue, Nov 12, 2019 at 5:03 AM leesf < leesf0315@gmail.com> wrote:
>>
>> [1] +1. `views` indeed confused me a lot.
>> [2] +1. `snapshot` is more reasonable.
>> [3] I don't feel very strong to rename it, the current name
>>
>> `COPY_ON_WRITE`
>>
>> is reasonable considering the cost to rename and the behavior that new
>> version parquet file will be created and seems to be copied from old
>> version parquet file.
>>
>> Best,
>> Leesf
>>
>> Balaji Varadarajan < vbalaji@apache.org> 于2019年11月12日周二 下午3:55写道：
>>
>> Agree with all 3 changes. The naming now looks more consistent than
>> earlier. +1 on them
>>
>> Depending on whether we are renaming Input formats for (1) and (2) -
>>
>> this
>>
>> could require some migration steps for
>>
>> Balaji.V
>>
>> >
>>
>> On Mon, Nov 11, 2019 at 7:38 PM vino yang < yanghua1127@gmail.com>
>>
>> wrote:
>>
>> Hi Vinoth,
>>
>> Thanks for bringing these proposals.
>>
>> +1 on all three. Especially, big +1 on the third renaming proposal.
>>
>> When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
>>
>> easily
>>
>> mislead users on the "copy" term. And make users compare it with
>>
>> the
>>
>> `CopyOnWriteArrayList` data structure provided by JDK and thoughts
>>
>> of
>>
>> the
>>
>> file systems.
>>
>> Best,
>> Vino
>>
>> >
>>
>> Bhavani Sudha < bhavanisudhas@gmail.com> 于2019年11月12日周二 上午9:05写道：
>>
>> +1 on all three rename proposals. I think this would make the
>>
>> concepts
>>
>> super easy to follow for new users.
>>
>> If changing [3] seems to be a stretch, we should definitely do
>>
>> [1]
>>
>> &
>>
>> [2]
>>
>> at
>>
>> the least IMO. I will be glad to help out on the renames to
>>
>> whatever
>>
>> extent
>>
>> possible should the Hudi community incline to pursue this.
>>
>> Thanks,
>> Sudha
>>
>> >
>> >
>>
>> On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <
>>
>> vinoth@apache.org>
>>
>> wrote:
>> >
>>
>> Hello all,
>>
>> I wanted to raise an important topic with the community around
>>
>> whether
>>
>> we
>>
>> should rename some of our terminologies in code/docs to be more
>> user-friendly and understandable..
>>
>> Let me also provide some context for each, since I am probably
>>
>> guilty
>>
>> of
>>
>> introducing most of them in the first place :).
>>
>> *1. Rename "views" to "query" : *Instead of saying incremental
>>
>> view
>>
>> or
>>
>> read-optimized view, talk about them as "incremental query" and
>> "read-optimized query". The term "view" is very technical, and
>>
>> what I
>>
>> was
>>
>> trying to convey was that we ingest/store the data once and
>>
>> expose
>>
>> views
>>
>> on
>>
>> top. But new users (atleast half dozen of them to me) tend to
>>
>> confuse
>>
>> this
>>
>> with views/materialized views found in databases. Almost always
>>
>> we
>>
>> talk
>>
>> about views mostly in terms of expected behavior for a query on
>>
>> the
>>
>> view. I
>>
>> am proposing to just call these different query types since
>>
>> its a
>>
>> more
>>
>> universally accepted terminology and IMO clearer.
>>
>> *2. Rename "Read-Optimized/Realtime" views to Snapshot views +
>>
>> Have
>>
>> Read-Optimized view only for MOR storage :* This one is
>>
>> probably
>>
>> the
>>
>> trickiest. Hudi was always designed with MOR in mind, even as
>>
>> we
>>
>> were
>>
>> working on COW storage and consequently we named the pure
>>
>> parquet
>>
>> backed
>>
>> view as Read-Optimized, hoping to name parquet + avro based
>>
>> view
>>
>> as
>>
>> Write-Optimized. However, we opted to name it Realtime to
>>
>> emphasize
>>
>> the
>>
>> data freshness aspect. In retrospect, the views should have not
>>
>> been
>>
>> named
>>
>> after their performance characteristics but rather the classes
>>
>> of
>>
>> queries
>>
>> done on them and guarantees for those (point above #1).
>>
>> Moreover,
>>
>> once
>>
>> we
>>
>> have parquet embedded into the log format, then the tradeoffs
>>
>> may
>>
>> not
>>
>> be
>>
>> the same anyways.
>>
>> So combining with the renaming proposed in #1, we would end up
>>
>> with
>>
>> the
>>
>> following..
>>
>> Copy-On-Write :
>> [Old] Read-Optimized View => [New] Snapshot Query
>> [Old] Incremental View => [New] Incremental Query
>>
>> Merge-On-Read:
>> [Old] Realtime View => [New] Snapshot Query
>> [Old] Incremental View => [New] Incremental Query
>> [Old] ReadOptimzied View => [New] Read-Optimized Query (since
>>
>> it
>>
>> is
>>
>> read
>>
>> optimized compared to Snapshot query always, at the cost of
>>
>> staler
>>
>> data)
>>
>> Both changes #1 & #2 could be simpler changes to just code
>>
>> references,
>>
>> docs
>>
>> and configs.. we can support both string for sometime and
>>
>> deprecate
>>
>> eventually since queries are stateless.
>>
>> *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated
>>
>> since
>>
>> the
>>
>> design was very similar to
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Copy-2Don-2Dwrite&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=y9XF8-75xzGHY4yCbfVVWcIC1sbEXDxitqeAS2A6GoQ&e=
>>
>> filesystems
>> & snapshotting and we once hoped to push some of this logic
>>
>> into
>>
>> the
>>
>> storage itself, all in vain. but the name stuck, even though
>>
>> once
>>
>> we
>>
>> had
>>
>> MERGE_ON_READ the focus was often on merge costs etc, which the
>>
>> name
>>
>> COPY_ON_WRITE does not convey directly. I don't feel very
>>
>> strong
>>
>> about
>>
>> this
>>
>> and there is also cost to changing this since its persisted
>>
>> inside
>>
>>
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__hoodie.properties&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=930ugGMXsrqzE-acg9nfeoePBmVjTRG3gD765ihEiqU&e=
>>
>> and we will support both strings internally in
>>
>> code
>>
>> for
>>
>> backwards compatibility anyway
>>
>> Naming something is very hard (yes, try :)).I believe these
>>
>> changes
>>
>> will
>>
>> make the project simpler to understand for everyone out there.
>>
>> We
>>
>> also
>>
>> have
>>
>> tons of new people here, so I am also happy to let go, if its
>>
>> already
>>
>> clear
>>
>> :)
>>
>> Please use the bullet number when you share your feedback so we
>>
>> know
>>
>> what
>>
>> the discussion is about.
>>
>> Thanks
>> Vinoth
>>
>>

Re: [DISCUSS] Simplification of terminologies

[Vinoth Chandar <vi...@apache.org>]

Will review the POC in cwiki.  +1

Based on this feedback, I will proceed with the changes. Thanks all!



On Tue, Nov 12, 2019 at 10:47 PM Semantic Beeng <ni...@semanticbeeng.com>
wrote:

> @vc, I think of it as elaborating the #ubiquitouslanguage in DDD.
> See private email with references to a small POC in wiki and decide how to
> proceed.
>
> On November 12, 2019 at 10:04 PM Vinoth Chandar < vinoth@apache.org>
> wrote:
>
>
> Thanks everyone for the feedback. Looks like we are in general agreement.
>
> I am inclined to just do 1 & 2 and leave COPY_ON_WRITE as is based on
> great
> points Ethan and Shiyan raised. Makes sense..
> Will wait for 1-2 days still to close this thread.
>
> @semanticbeeing Thats a great idea. Is it more like a technical glossary
> of
> sorts? Lets may be start a different DISCUSS thread on that specific
> topic,
> so everyone can chime in and provide more attention to that proposal?
>
>
>
>
>
> On Tue, Nov 12, 2019 at 2:44 PM Y. Ethan Guo < guoyihua@uber.com.invalid>
> wrote:
>
> +1 on [1] and [2].
>
> For [3], I have similar doubts as Shiyan.
>
> For the naming, I can understand the original intent of the analogy for
> COW
> which is to make another "copy" of columnar/parquet file upon the
> modification/update to the records in the file. From the system design
> point of view, it's easy to understand. I'm okay with the renaming as
> "MERGE_ON_WRITE" since it's probably straightforward for users at the
> first
> glance.
>
> In terms of the concept, COW and MOR are listed as storage/table types.
> From my understanding, they represent different tradeoffs of the
> performance between reading and writing Hudi tables, and within MOR there
> are different tradeoffs, e.g., lazy merge on read or periodic compaction
> and cleaning pipelined along ingestion. It looks like these can be
> controlled through configs, e.g., "disable_merge_on_write",
> "compaction_frenquency", etc., instead of fixing the storage type, to
> control the tradeoff that a user would like to make. The requirement may
> change so a user can switch between COW and MOR by tuning the configs. We
> don't have to make such changes now, but I'm wondering if this is
> something
> worth considering in the future releases.
>
> - Ethan
>
> On Tue, Nov 12, 2019 at 8:43 AM nishith agarwal < n3.nash29@gmail.com>
> wrote:
>
> +1 on the first two, don't feel strongly about (3).
>
> Thanks,
> Nishith
>
> On Tue, Nov 12, 2019 at 5:03 AM leesf < leesf0315@gmail.com> wrote:
>
> [1] +1. `views` indeed confused me a lot.
> [2] +1. `snapshot` is more reasonable.
> [3] I don't feel very strong to rename it, the current name
>
> `COPY_ON_WRITE`
>
> is reasonable considering the cost to rename and the behavior that new
> version parquet file will be created and seems to be copied from old
> version parquet file.
>
> Best,
> Leesf
>
> Balaji Varadarajan < vbalaji@apache.org> 于2019年11月12日周二 下午3:55写道：
>
> Agree with all 3 changes. The naming now looks more consistent than
> earlier. +1 on them
>
> Depending on whether we are renaming Input formats for (1) and (2) -
>
> this
>
> could require some migration steps for
>
> Balaji.V
>
> >
>
> On Mon, Nov 11, 2019 at 7:38 PM vino yang < yanghua1127@gmail.com>
>
> wrote:
>
> Hi Vinoth,
>
> Thanks for bringing these proposals.
>
> +1 on all three. Especially, big +1 on the third renaming proposal.
>
> When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
>
> easily
>
> mislead users on the "copy" term. And make users compare it with
>
> the
>
> `CopyOnWriteArrayList` data structure provided by JDK and thoughts
>
> of
>
> the
>
> file systems.
>
> Best,
> Vino
>
> >
>
> Bhavani Sudha < bhavanisudhas@gmail.com> 于2019年11月12日周二 上午9:05写道：
>
> +1 on all three rename proposals. I think this would make the
>
> concepts
>
> super easy to follow for new users.
>
> If changing [3] seems to be a stretch, we should definitely do
>
> [1]
>
> &
>
> [2]
>
> at
>
> the least IMO. I will be glad to help out on the renames to
>
> whatever
>
> extent
>
> possible should the Hudi community incline to pursue this.
>
> Thanks,
> Sudha
>
> >
> >
>
> On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <
>
> vinoth@apache.org>
>
> wrote:
> >
>
> Hello all,
>
> I wanted to raise an important topic with the community around
>
> whether
>
> we
>
> should rename some of our terminologies in code/docs to be more
> user-friendly and understandable..
>
> Let me also provide some context for each, since I am probably
>
> guilty
>
> of
>
> introducing most of them in the first place :).
>
> *1. Rename "views" to "query" : *Instead of saying incremental
>
> view
>
> or
>
> read-optimized view, talk about them as "incremental query" and
> "read-optimized query". The term "view" is very technical, and
>
> what I
>
> was
>
> trying to convey was that we ingest/store the data once and
>
> expose
>
> views
>
> on
>
> top. But new users (atleast half dozen of them to me) tend to
>
> confuse
>
> this
>
> with views/materialized views found in databases. Almost always
>
> we
>
> talk
>
> about views mostly in terms of expected behavior for a query on
>
> the
>
> view. I
>
> am proposing to just call these different query types since
>
> its a
>
> more
>
> universally accepted terminology and IMO clearer.
>
> *2. Rename "Read-Optimized/Realtime" views to Snapshot views +
>
> Have
>
> Read-Optimized view only for MOR storage :* This one is
>
> probably
>
> the
>
> trickiest. Hudi was always designed with MOR in mind, even as
>
> we
>
> were
>
> working on COW storage and consequently we named the pure
>
> parquet
>
> backed
>
> view as Read-Optimized, hoping to name parquet + avro based
>
> view
>
> as
>
> Write-Optimized. However, we opted to name it Realtime to
>
> emphasize
>
> the
>
> data freshness aspect. In retrospect, the views should have not
>
> been
>
> named
>
> after their performance characteristics but rather the classes
>
> of
>
> queries
>
> done on them and guarantees for those (point above #1).
>
> Moreover,
>
> once
>
> we
>
> have parquet embedded into the log format, then the tradeoffs
>
> may
>
> not
>
> be
>
> the same anyways.
>
> So combining with the renaming proposed in #1, we would end up
>
> with
>
> the
>
> following..
>
> Copy-On-Write :
> [Old] Read-Optimized View => [New] Snapshot Query
> [Old] Incremental View => [New] Incremental Query
>
> Merge-On-Read:
> [Old] Realtime View => [New] Snapshot Query
> [Old] Incremental View => [New] Incremental Query
> [Old] ReadOptimzied View => [New] Read-Optimized Query (since
>
> it
>
> is
>
> read
>
> optimized compared to Snapshot query always, at the cost of
>
> staler
>
> data)
>
> Both changes #1 & #2 could be simpler changes to just code
>
> references,
>
> docs
>
> and configs.. we can support both string for sometime and
>
> deprecate
>
> eventually since queries are stateless.
>
> *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated
>
> since
>
> the
>
> design was very similar to
>
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Copy-2Don-2Dwrite&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=y9XF8-75xzGHY4yCbfVVWcIC1sbEXDxitqeAS2A6GoQ&e=
>
> filesystems
> & snapshotting and we once hoped to push some of this logic
>
> into
>
> the
>
> storage itself, all in vain. but the name stuck, even though
>
> once
>
> we
>
> had
>
> MERGE_ON_READ the focus was often on merge costs etc, which the
>
> name
>
> COPY_ON_WRITE does not convey directly. I don't feel very
>
> strong
>
> about
>
> this
>
> and there is also cost to changing this since its persisted
>
> inside
>
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__hoodie.properties&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=930ugGMXsrqzE-acg9nfeoePBmVjTRG3gD765ihEiqU&e=
>
> and we will support both strings internally in
>
> code
>
> for
>
> backwards compatibility anyway
>
> Naming something is very hard (yes, try :)).I believe these
>
> changes
>
> will
>
> make the project simpler to understand for everyone out there.
>
> We
>
> also
>
> have
>
> tons of new people here, so I am also happy to let go, if its
>
> already
>
> clear
>
> :)
>
> Please use the bullet number when you share your feedback so we
>
> know
>
> what
>
> the discussion is about.
>
> Thanks
> Vinoth
>
>

Re: [DISCUSS] Simplification of terminologies

[Vinoth Chandar <vi...@apache.org>]

Thanks everyone for the feedback. Looks like we are in general agreement.

I am inclined to just do 1 & 2 and leave COPY_ON_WRITE as is based on great
points Ethan and Shiyan raised. Makes sense..
Will wait for 1-2 days still to close this thread.

@semanticbeeing Thats a great idea. Is it more like a technical glossary of
sorts? Lets may be start a different DISCUSS thread on that specific topic,
so everyone can chime in and provide more attention to that proposal?





On Tue, Nov 12, 2019 at 2:44 PM Y. Ethan Guo <gu...@uber.com.invalid>
wrote:

> +1 on [1] and [2].
>
> For [3], I have similar doubts as Shiyan.
>
> For the naming, I can understand the original intent of the analogy for COW
> which is to make another "copy" of columnar/parquet file upon the
> modification/update to the records in the file.  From the system design
> point of view, it's easy to understand.  I'm okay with the renaming as
> "MERGE_ON_WRITE" since it's probably straightforward for users at the first
> glance.
>
> In terms of the concept, COW and MOR are listed as storage/table types.
> From my understanding, they represent different tradeoffs of the
> performance between reading and writing Hudi tables, and within MOR there
> are different tradeoffs, e.g., lazy merge on read or periodic compaction
> and cleaning pipelined along ingestion. It looks like these can be
> controlled through configs, e.g., "disable_merge_on_write",
> "compaction_frenquency", etc., instead of fixing the storage type, to
> control the tradeoff that a user would like to make.  The requirement may
> change so a user can switch between COW and MOR by tuning the configs. We
> don't have to make such changes now, but I'm wondering if this is something
> worth considering in the future releases.
>
> - Ethan
>
> On Tue, Nov 12, 2019 at 8:43 AM nishith agarwal <n3...@gmail.com>
> wrote:
>
> > +1 on the first two, don't feel strongly about (3).
> >
> > Thanks,
> > Nishith
> >
> > On Tue, Nov 12, 2019 at 5:03 AM leesf <le...@gmail.com> wrote:
> >
> > > [1] +1. `views` indeed confused me a lot.
> > > [2] +1. `snapshot` is more reasonable.
> > > [3] I don't feel very strong to rename it, the current name
> > `COPY_ON_WRITE`
> > > is reasonable considering the cost to rename and the behavior that new
> > > version parquet file will be created and seems to be copied from old
> > > version parquet file.
> > >
> > > Best,
> > > Leesf
> > >
> > > Balaji Varadarajan <vb...@apache.org> 于2019年11月12日周二 下午3:55写道：
> > >
> > > > Agree with all 3 changes. The naming now looks more consistent than
> > > > earlier. +1 on them
> > > >
> > > > Depending on whether we are renaming Input formats for (1) and (2) -
> > this
> > > > could require some migration steps for
> > > >
> > > > Balaji.V
> > > >
> > > >
> > > > On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com>
> > wrote:
> > > >
> > > > > Hi Vinoth,
> > > > >
> > > > > Thanks for bringing these proposals.
> > > > >
> > > > > +1 on all three. Especially, big +1 on the third renaming proposal.
> > > > >
> > > > > When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
> > > > easily
> > > > > mislead users on the "copy" term. And make users compare it with
> the
> > > > > `CopyOnWriteArrayList` data structure provided by JDK  and thoughts
> > of
> > > > the
> > > > > file systems.
> > > > >
> > > > > Best,
> > > > > Vino
> > > > >
> > > > >
> > > > > Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
> > > > >
> > > > > > +1 on all three rename proposals. I think this would make the
> > > concepts
> > > > > > super easy to follow for new users.
> > > > > >
> > > > > > If changing [3] seems to be a stretch, we should definitely do
> [1]
> > &
> > > > [2]
> > > > > at
> > > > > > the least IMO. I will be glad to help out on the renames to
> > whatever
> > > > > extent
> > > > > > possible should the Hudi community incline to pursue this.
> > > > > >
> > > > > > Thanks,
> > > > > > Sudha
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <
> vinoth@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hello all,
> > > > > > >
> > > > > > > I wanted to raise an important topic with the community around
> > > > whether
> > > > > we
> > > > > > > should rename some of our terminologies in code/docs to be more
> > > > > > > user-friendly and understandable..
> > > > > > >
> > > > > > > Let me also provide some context for each, since I am probably
> > > guilty
> > > > > of
> > > > > > > introducing most of them in the first place :).
> > > > > > >
> > > > > > > *1. Rename "views" to "query" : *Instead of saying incremental
> > view
> > > > or
> > > > > > > read-optimized view, talk about them as "incremental query" and
> > > > > > > "read-optimized query". The term "view" is very technical, and
> > > what I
> > > > > was
> > > > > > > trying to convey was that we ingest/store the data once and
> > expose
> > > > > views
> > > > > > on
> > > > > > > top. But new users (atleast half dozen of them to me) tend to
> > > confuse
> > > > > > this
> > > > > > > with views/materialized views found in databases. Almost always
> > we
> > > > talk
> > > > > > > about views mostly in terms of expected behavior for a query on
> > the
> > > > > > view. I
> > > > > > > am proposing to just call these different query types since
> its a
> > > > more
> > > > > > > universally accepted terminology and IMO clearer.
> > > > > > >
> > > > > > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views +
> > Have
> > > > > > > Read-Optimized view only for MOR storage :* This one is
> probably
> > > the
> > > > > > > trickiest. Hudi was always designed with MOR in mind, even as
> we
> > > were
> > > > > > > working on COW storage and consequently we named the pure
> parquet
> > > > > backed
> > > > > > > view as Read-Optimized, hoping to name parquet + avro based
> view
> > as
> > > > > > > Write-Optimized. However, we opted to name it Realtime to
> > emphasize
> > > > the
> > > > > > > data freshness aspect. In retrospect, the views should have not
> > > been
> > > > > > named
> > > > > > > after their performance characteristics but rather the classes
> of
> > > > > queries
> > > > > > > done on them and guarantees for those (point above #1).
> Moreover,
> > > > once
> > > > > we
> > > > > > > have parquet embedded into the log format, then the tradeoffs
> may
> > > not
> > > > > be
> > > > > > > the same anyways.
> > > > > > >
> > > > > > > So combining with the renaming proposed in #1, we would end up
> > with
> > > > the
> > > > > > > following..
> > > > > > >
> > > > > > > Copy-On-Write :
> > > > > > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > > > > > [Old]  Incremental View => [New] Incremental Query
> > > > > > >
> > > > > > > Merge-On-Read:
> > > > > > > [Old] Realtime View => [New] Snapshot Query
> > > > > > > [Old] Incremental View => [New] Incremental Query
> > > > > > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since
> it
> > is
> > > > > read
> > > > > > > optimized compared to Snapshot query always, at the cost of
> > staler
> > > > > data)
> > > > > > >
> > > > > > > Both changes #1 & #2 could be simpler changes to just code
> > > > references,
> > > > > > docs
> > > > > > > and configs.. we can support both string for sometime and
> > deprecate
> > > > > > > eventually since queries are stateless.
> > > > > > >
> > > > > > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated
> > since
> > > > the
> > > > > > > design was very similar to
> > > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Copy-2Don-2Dwrite&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=y9XF8-75xzGHY4yCbfVVWcIC1sbEXDxitqeAS2A6GoQ&e=
> > > > > > > filesystems
> > > > > > > & snapshotting and we once hoped to push some of this logic
> into
> > > the
> > > > > > > storage itself, all in vain. but the name stuck, even though
> once
> > > we
> > > > > had
> > > > > > > MERGE_ON_READ the focus was often on merge costs etc, which the
> > > name
> > > > > > > COPY_ON_WRITE does not convey directly. I don't feel very
> strong
> > > > about
> > > > > > this
> > > > > > > and there is also cost to changing this since its persisted
> > inside
> > > > > > >
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__hoodie.properties&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=930ugGMXsrqzE-acg9nfeoePBmVjTRG3gD765ihEiqU&e=
> > and we will support both strings internally in
> > > code
> > > > > for
> > > > > > > backwards compatibility anyway
> > > > > > >
> > > > > > > Naming something is very hard (yes, try :)).I believe these
> > changes
> > > > > will
> > > > > > > make the project simpler to understand for everyone out there.
> We
> > > > also
> > > > > > have
> > > > > > > tons of new people here, so I am also happy to let go, if its
> > > already
> > > > > > clear
> > > > > > > :)
> > > > > > >
> > > > > > > Please use the bullet number when you share your feedback so we
> > > know
> > > > > what
> > > > > > > the discussion is about.
> > > > > > >
> > > > > > > Thanks
> > > > > > > Vinoth
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

["Y. Ethan Guo" <gu...@uber.com.INVALID>]

+1 on [1] and [2].

For [3], I have similar doubts as Shiyan.

For the naming, I can understand the original intent of the analogy for COW
which is to make another "copy" of columnar/parquet file upon the
modification/update to the records in the file.  From the system design
point of view, it's easy to understand.  I'm okay with the renaming as
"MERGE_ON_WRITE" since it's probably straightforward for users at the first
glance.

In terms of the concept, COW and MOR are listed as storage/table types.
From my understanding, they represent different tradeoffs of the
performance between reading and writing Hudi tables, and within MOR there
are different tradeoffs, e.g., lazy merge on read or periodic compaction
and cleaning pipelined along ingestion. It looks like these can be
controlled through configs, e.g., "disable_merge_on_write",
"compaction_frenquency", etc., instead of fixing the storage type, to
control the tradeoff that a user would like to make.  The requirement may
change so a user can switch between COW and MOR by tuning the configs. We
don't have to make such changes now, but I'm wondering if this is something
worth considering in the future releases.

- Ethan

On Tue, Nov 12, 2019 at 8:43 AM nishith agarwal <n3...@gmail.com> wrote:

> +1 on the first two, don't feel strongly about (3).
>
> Thanks,
> Nishith
>
> On Tue, Nov 12, 2019 at 5:03 AM leesf <le...@gmail.com> wrote:
>
> > [1] +1. `views` indeed confused me a lot.
> > [2] +1. `snapshot` is more reasonable.
> > [3] I don't feel very strong to rename it, the current name
> `COPY_ON_WRITE`
> > is reasonable considering the cost to rename and the behavior that new
> > version parquet file will be created and seems to be copied from old
> > version parquet file.
> >
> > Best,
> > Leesf
> >
> > Balaji Varadarajan <vb...@apache.org> 于2019年11月12日周二 下午3:55写道：
> >
> > > Agree with all 3 changes. The naming now looks more consistent than
> > > earlier. +1 on them
> > >
> > > Depending on whether we are renaming Input formats for (1) and (2) -
> this
> > > could require some migration steps for
> > >
> > > Balaji.V
> > >
> > >
> > > On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com>
> wrote:
> > >
> > > > Hi Vinoth,
> > > >
> > > > Thanks for bringing these proposals.
> > > >
> > > > +1 on all three. Especially, big +1 on the third renaming proposal.
> > > >
> > > > When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
> > > easily
> > > > mislead users on the "copy" term. And make users compare it with the
> > > > `CopyOnWriteArrayList` data structure provided by JDK  and thoughts
> of
> > > the
> > > > file systems.
> > > >
> > > > Best,
> > > > Vino
> > > >
> > > >
> > > > Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
> > > >
> > > > > +1 on all three rename proposals. I think this would make the
> > concepts
> > > > > super easy to follow for new users.
> > > > >
> > > > > If changing [3] seems to be a stretch, we should definitely do [1]
> &
> > > [2]
> > > > at
> > > > > the least IMO. I will be glad to help out on the renames to
> whatever
> > > > extent
> > > > > possible should the Hudi community incline to pursue this.
> > > > >
> > > > > Thanks,
> > > > > Sudha
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org>
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I wanted to raise an important topic with the community around
> > > whether
> > > > we
> > > > > > should rename some of our terminologies in code/docs to be more
> > > > > > user-friendly and understandable..
> > > > > >
> > > > > > Let me also provide some context for each, since I am probably
> > guilty
> > > > of
> > > > > > introducing most of them in the first place :).
> > > > > >
> > > > > > *1. Rename "views" to "query" : *Instead of saying incremental
> view
> > > or
> > > > > > read-optimized view, talk about them as "incremental query" and
> > > > > > "read-optimized query". The term "view" is very technical, and
> > what I
> > > > was
> > > > > > trying to convey was that we ingest/store the data once and
> expose
> > > > views
> > > > > on
> > > > > > top. But new users (atleast half dozen of them to me) tend to
> > confuse
> > > > > this
> > > > > > with views/materialized views found in databases. Almost always
> we
> > > talk
> > > > > > about views mostly in terms of expected behavior for a query on
> the
> > > > > view. I
> > > > > > am proposing to just call these different query types since its a
> > > more
> > > > > > universally accepted terminology and IMO clearer.
> > > > > >
> > > > > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views +
> Have
> > > > > > Read-Optimized view only for MOR storage :* This one is probably
> > the
> > > > > > trickiest. Hudi was always designed with MOR in mind, even as we
> > were
> > > > > > working on COW storage and consequently we named the pure parquet
> > > > backed
> > > > > > view as Read-Optimized, hoping to name parquet + avro based view
> as
> > > > > > Write-Optimized. However, we opted to name it Realtime to
> emphasize
> > > the
> > > > > > data freshness aspect. In retrospect, the views should have not
> > been
> > > > > named
> > > > > > after their performance characteristics but rather the classes of
> > > > queries
> > > > > > done on them and guarantees for those (point above #1). Moreover,
> > > once
> > > > we
> > > > > > have parquet embedded into the log format, then the tradeoffs may
> > not
> > > > be
> > > > > > the same anyways.
> > > > > >
> > > > > > So combining with the renaming proposed in #1, we would end up
> with
> > > the
> > > > > > following..
> > > > > >
> > > > > > Copy-On-Write :
> > > > > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > > > > [Old]  Incremental View => [New] Incremental Query
> > > > > >
> > > > > > Merge-On-Read:
> > > > > > [Old] Realtime View => [New] Snapshot Query
> > > > > > [Old] Incremental View => [New] Incremental Query
> > > > > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it
> is
> > > > read
> > > > > > optimized compared to Snapshot query always, at the cost of
> staler
> > > > data)
> > > > > >
> > > > > > Both changes #1 & #2 could be simpler changes to just code
> > > references,
> > > > > docs
> > > > > > and configs.. we can support both string for sometime and
> deprecate
> > > > > > eventually since queries are stateless.
> > > > > >
> > > > > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated
> since
> > > the
> > > > > > design was very similar to
> > >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Copy-2Don-2Dwrite&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=y9XF8-75xzGHY4yCbfVVWcIC1sbEXDxitqeAS2A6GoQ&e=
> > > > > > filesystems
> > > > > > & snapshotting and we once hoped to push some of this logic into
> > the
> > > > > > storage itself, all in vain. but the name stuck, even though once
> > we
> > > > had
> > > > > > MERGE_ON_READ the focus was often on merge costs etc, which the
> > name
> > > > > > COPY_ON_WRITE does not convey directly. I don't feel very strong
> > > about
> > > > > this
> > > > > > and there is also cost to changing this since its persisted
> inside
> > > > > >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__hoodie.properties&d=DwIFaQ&c=r2dcLCtU9q6n0vrtnDw9vg&r=z456dQQXMUCz1m72nlkFQpylUpdOVMBG38x2peG1m44&m=m1yKGEwnAUe_FyIsWFAo-YVKyfq1nayItNGNc7iv8Yw&s=930ugGMXsrqzE-acg9nfeoePBmVjTRG3gD765ihEiqU&e=
> and we will support both strings internally in
> > code
> > > > for
> > > > > > backwards compatibility anyway
> > > > > >
> > > > > > Naming something is very hard (yes, try :)).I believe these
> changes
> > > > will
> > > > > > make the project simpler to understand for everyone out there. We
> > > also
> > > > > have
> > > > > > tons of new people here, so I am also happy to let go, if its
> > already
> > > > > clear
> > > > > > :)
> > > > > >
> > > > > > Please use the bullet number when you share your feedback so we
> > know
> > > > what
> > > > > > the discussion is about.
> > > > > >
> > > > > > Thanks
> > > > > > Vinoth
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

[nishith agarwal <n3...@gmail.com>]

+1 on the first two, don't feel strongly about (3).

Thanks,
Nishith

On Tue, Nov 12, 2019 at 5:03 AM leesf <le...@gmail.com> wrote:

> [1] +1. `views` indeed confused me a lot.
> [2] +1. `snapshot` is more reasonable.
> [3] I don't feel very strong to rename it, the current name `COPY_ON_WRITE`
> is reasonable considering the cost to rename and the behavior that new
> version parquet file will be created and seems to be copied from old
> version parquet file.
>
> Best,
> Leesf
>
> Balaji Varadarajan <vb...@apache.org> 于2019年11月12日周二 下午3:55写道：
>
> > Agree with all 3 changes. The naming now looks more consistent than
> > earlier. +1 on them
> >
> > Depending on whether we are renaming Input formats for (1) and (2) - this
> > could require some migration steps for
> >
> > Balaji.V
> >
> >
> > On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com> wrote:
> >
> > > Hi Vinoth,
> > >
> > > Thanks for bringing these proposals.
> > >
> > > +1 on all three. Especially, big +1 on the third renaming proposal.
> > >
> > > When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
> > easily
> > > mislead users on the "copy" term. And make users compare it with the
> > > `CopyOnWriteArrayList` data structure provided by JDK  and thoughts of
> > the
> > > file systems.
> > >
> > > Best,
> > > Vino
> > >
> > >
> > > Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
> > >
> > > > +1 on all three rename proposals. I think this would make the
> concepts
> > > > super easy to follow for new users.
> > > >
> > > > If changing [3] seems to be a stretch, we should definitely do [1] &
> > [2]
> > > at
> > > > the least IMO. I will be glad to help out on the renames to whatever
> > > extent
> > > > possible should the Hudi community incline to pursue this.
> > > >
> > > > Thanks,
> > > > Sudha
> > > >
> > > >
> > > >
> > > > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org>
> > > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I wanted to raise an important topic with the community around
> > whether
> > > we
> > > > > should rename some of our terminologies in code/docs to be more
> > > > > user-friendly and understandable..
> > > > >
> > > > > Let me also provide some context for each, since I am probably
> guilty
> > > of
> > > > > introducing most of them in the first place :).
> > > > >
> > > > > *1. Rename "views" to "query" : *Instead of saying incremental view
> > or
> > > > > read-optimized view, talk about them as "incremental query" and
> > > > > "read-optimized query". The term "view" is very technical, and
> what I
> > > was
> > > > > trying to convey was that we ingest/store the data once and expose
> > > views
> > > > on
> > > > > top. But new users (atleast half dozen of them to me) tend to
> confuse
> > > > this
> > > > > with views/materialized views found in databases. Almost always we
> > talk
> > > > > about views mostly in terms of expected behavior for a query on the
> > > > view. I
> > > > > am proposing to just call these different query types since its a
> > more
> > > > > universally accepted terminology and IMO clearer.
> > > > >
> > > > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> > > > > Read-Optimized view only for MOR storage :* This one is probably
> the
> > > > > trickiest. Hudi was always designed with MOR in mind, even as we
> were
> > > > > working on COW storage and consequently we named the pure parquet
> > > backed
> > > > > view as Read-Optimized, hoping to name parquet + avro based view as
> > > > > Write-Optimized. However, we opted to name it Realtime to emphasize
> > the
> > > > > data freshness aspect. In retrospect, the views should have not
> been
> > > > named
> > > > > after their performance characteristics but rather the classes of
> > > queries
> > > > > done on them and guarantees for those (point above #1). Moreover,
> > once
> > > we
> > > > > have parquet embedded into the log format, then the tradeoffs may
> not
> > > be
> > > > > the same anyways.
> > > > >
> > > > > So combining with the renaming proposed in #1, we would end up with
> > the
> > > > > following..
> > > > >
> > > > > Copy-On-Write :
> > > > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > > > [Old]  Incremental View => [New] Incremental Query
> > > > >
> > > > > Merge-On-Read:
> > > > > [Old] Realtime View => [New] Snapshot Query
> > > > > [Old] Incremental View => [New] Incremental Query
> > > > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is
> > > read
> > > > > optimized compared to Snapshot query always, at the cost of staler
> > > data)
> > > > >
> > > > > Both changes #1 & #2 could be simpler changes to just code
> > references,
> > > > docs
> > > > > and configs.. we can support both string for sometime and deprecate
> > > > > eventually since queries are stateless.
> > > > >
> > > > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since
> > the
> > > > > design was very similar to
> > https://en.wikipedia.org/wiki/Copy-on-write
> > > > > filesystems
> > > > > & snapshotting and we once hoped to push some of this logic into
> the
> > > > > storage itself, all in vain. but the name stuck, even though once
> we
> > > had
> > > > > MERGE_ON_READ the focus was often on merge costs etc, which the
> name
> > > > > COPY_ON_WRITE does not convey directly. I don't feel very strong
> > about
> > > > this
> > > > > and there is also cost to changing this since its persisted inside
> > > > > hoodie.properties and we will support both strings internally in
> code
> > > for
> > > > > backwards compatibility anyway
> > > > >
> > > > > Naming something is very hard (yes, try :)).I believe these changes
> > > will
> > > > > make the project simpler to understand for everyone out there. We
> > also
> > > > have
> > > > > tons of new people here, so I am also happy to let go, if its
> already
> > > > clear
> > > > > :)
> > > > >
> > > > > Please use the bullet number when you share your feedback so we
> know
> > > what
> > > > > the discussion is about.
> > > > >
> > > > > Thanks
> > > > > Vinoth
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

[leesf <le...@gmail.com>]

[1] +1. `views` indeed confused me a lot.
[2] +1. `snapshot` is more reasonable.
[3] I don't feel very strong to rename it, the current name `COPY_ON_WRITE`
is reasonable considering the cost to rename and the behavior that new
version parquet file will be created and seems to be copied from old
version parquet file.

Best,
Leesf

Balaji Varadarajan <vb...@apache.org> 于2019年11月12日周二 下午3:55写道：

> Agree with all 3 changes. The naming now looks more consistent than
> earlier. +1 on them
>
> Depending on whether we are renaming Input formats for (1) and (2) - this
> could require some migration steps for
>
> Balaji.V
>
>
> On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com> wrote:
>
> > Hi Vinoth,
> >
> > Thanks for bringing these proposals.
> >
> > +1 on all three. Especially, big +1 on the third renaming proposal.
> >
> > When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It
> easily
> > mislead users on the "copy" term. And make users compare it with the
> > `CopyOnWriteArrayList` data structure provided by JDK  and thoughts of
> the
> > file systems.
> >
> > Best,
> > Vino
> >
> >
> > Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
> >
> > > +1 on all three rename proposals. I think this would make the concepts
> > > super easy to follow for new users.
> > >
> > > If changing [3] seems to be a stretch, we should definitely do [1] &
> [2]
> > at
> > > the least IMO. I will be glad to help out on the renames to whatever
> > extent
> > > possible should the Hudi community incline to pursue this.
> > >
> > > Thanks,
> > > Sudha
> > >
> > >
> > >
> > > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org>
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I wanted to raise an important topic with the community around
> whether
> > we
> > > > should rename some of our terminologies in code/docs to be more
> > > > user-friendly and understandable..
> > > >
> > > > Let me also provide some context for each, since I am probably guilty
> > of
> > > > introducing most of them in the first place :).
> > > >
> > > > *1. Rename "views" to "query" : *Instead of saying incremental view
> or
> > > > read-optimized view, talk about them as "incremental query" and
> > > > "read-optimized query". The term "view" is very technical, and what I
> > was
> > > > trying to convey was that we ingest/store the data once and expose
> > views
> > > on
> > > > top. But new users (atleast half dozen of them to me) tend to confuse
> > > this
> > > > with views/materialized views found in databases. Almost always we
> talk
> > > > about views mostly in terms of expected behavior for a query on the
> > > view. I
> > > > am proposing to just call these different query types since its a
> more
> > > > universally accepted terminology and IMO clearer.
> > > >
> > > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> > > > Read-Optimized view only for MOR storage :* This one is probably the
> > > > trickiest. Hudi was always designed with MOR in mind, even as we were
> > > > working on COW storage and consequently we named the pure parquet
> > backed
> > > > view as Read-Optimized, hoping to name parquet + avro based view as
> > > > Write-Optimized. However, we opted to name it Realtime to emphasize
> the
> > > > data freshness aspect. In retrospect, the views should have not been
> > > named
> > > > after their performance characteristics but rather the classes of
> > queries
> > > > done on them and guarantees for those (point above #1). Moreover,
> once
> > we
> > > > have parquet embedded into the log format, then the tradeoffs may not
> > be
> > > > the same anyways.
> > > >
> > > > So combining with the renaming proposed in #1, we would end up with
> the
> > > > following..
> > > >
> > > > Copy-On-Write :
> > > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > > [Old]  Incremental View => [New] Incremental Query
> > > >
> > > > Merge-On-Read:
> > > > [Old] Realtime View => [New] Snapshot Query
> > > > [Old] Incremental View => [New] Incremental Query
> > > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is
> > read
> > > > optimized compared to Snapshot query always, at the cost of staler
> > data)
> > > >
> > > > Both changes #1 & #2 could be simpler changes to just code
> references,
> > > docs
> > > > and configs.. we can support both string for sometime and deprecate
> > > > eventually since queries are stateless.
> > > >
> > > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since
> the
> > > > design was very similar to
> https://en.wikipedia.org/wiki/Copy-on-write
> > > > filesystems
> > > > & snapshotting and we once hoped to push some of this logic into the
> > > > storage itself, all in vain. but the name stuck, even though once we
> > had
> > > > MERGE_ON_READ the focus was often on merge costs etc, which the name
> > > > COPY_ON_WRITE does not convey directly. I don't feel very strong
> about
> > > this
> > > > and there is also cost to changing this since its persisted inside
> > > > hoodie.properties and we will support both strings internally in code
> > for
> > > > backwards compatibility anyway
> > > >
> > > > Naming something is very hard (yes, try :)).I believe these changes
> > will
> > > > make the project simpler to understand for everyone out there. We
> also
> > > have
> > > > tons of new people here, so I am also happy to let go, if its already
> > > clear
> > > > :)
> > > >
> > > > Please use the bullet number when you share your feedback so we know
> > what
> > > > the discussion is about.
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

[Balaji Varadarajan <vb...@apache.org>]

Agree with all 3 changes. The naming now looks more consistent than
earlier. +1 on them

Depending on whether we are renaming Input formats for (1) and (2) - this
could require some migration steps for

Balaji.V


On Mon, Nov 11, 2019 at 7:38 PM vino yang <ya...@gmail.com> wrote:

> Hi Vinoth,
>
> Thanks for bringing these proposals.
>
> +1 on all three. Especially, big +1 on the third renaming proposal.
>
> When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It easily
> mislead users on the "copy" term. And make users compare it with the
> `CopyOnWriteArrayList` data structure provided by JDK  and thoughts of the
> file systems.
>
> Best,
> Vino
>
>
> Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：
>
> > +1 on all three rename proposals. I think this would make the concepts
> > super easy to follow for new users.
> >
> > If changing [3] seems to be a stretch, we should definitely do [1] & [2]
> at
> > the least IMO. I will be glad to help out on the renames to whatever
> extent
> > possible should the Hudi community incline to pursue this.
> >
> > Thanks,
> > Sudha
> >
> >
> >
> > On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org>
> wrote:
> >
> > > Hello all,
> > >
> > > I wanted to raise an important topic with the community around whether
> we
> > > should rename some of our terminologies in code/docs to be more
> > > user-friendly and understandable..
> > >
> > > Let me also provide some context for each, since I am probably guilty
> of
> > > introducing most of them in the first place :).
> > >
> > > *1. Rename "views" to "query" : *Instead of saying incremental view or
> > > read-optimized view, talk about them as "incremental query" and
> > > "read-optimized query". The term "view" is very technical, and what I
> was
> > > trying to convey was that we ingest/store the data once and expose
> views
> > on
> > > top. But new users (atleast half dozen of them to me) tend to confuse
> > this
> > > with views/materialized views found in databases. Almost always we talk
> > > about views mostly in terms of expected behavior for a query on the
> > view. I
> > > am proposing to just call these different query types since its a more
> > > universally accepted terminology and IMO clearer.
> > >
> > > *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> > > Read-Optimized view only for MOR storage :* This one is probably the
> > > trickiest. Hudi was always designed with MOR in mind, even as we were
> > > working on COW storage and consequently we named the pure parquet
> backed
> > > view as Read-Optimized, hoping to name parquet + avro based view as
> > > Write-Optimized. However, we opted to name it Realtime to emphasize the
> > > data freshness aspect. In retrospect, the views should have not been
> > named
> > > after their performance characteristics but rather the classes of
> queries
> > > done on them and guarantees for those (point above #1). Moreover, once
> we
> > > have parquet embedded into the log format, then the tradeoffs may not
> be
> > > the same anyways.
> > >
> > > So combining with the renaming proposed in #1, we would end up with the
> > > following..
> > >
> > > Copy-On-Write :
> > > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > > [Old]  Incremental View => [New] Incremental Query
> > >
> > > Merge-On-Read:
> > > [Old] Realtime View => [New] Snapshot Query
> > > [Old] Incremental View => [New] Incremental Query
> > > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is
> read
> > > optimized compared to Snapshot query always, at the cost of staler
> data)
> > >
> > > Both changes #1 & #2 could be simpler changes to just code references,
> > docs
> > > and configs.. we can support both string for sometime and deprecate
> > > eventually since queries are stateless.
> > >
> > > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since the
> > > design was very similar to https://en.wikipedia.org/wiki/Copy-on-write
> > > filesystems
> > > & snapshotting and we once hoped to push some of this logic into the
> > > storage itself, all in vain. but the name stuck, even though once we
> had
> > > MERGE_ON_READ the focus was often on merge costs etc, which the name
> > > COPY_ON_WRITE does not convey directly. I don't feel very strong about
> > this
> > > and there is also cost to changing this since its persisted inside
> > > hoodie.properties and we will support both strings internally in code
> for
> > > backwards compatibility anyway
> > >
> > > Naming something is very hard (yes, try :)).I believe these changes
> will
> > > make the project simpler to understand for everyone out there. We also
> > have
> > > tons of new people here, so I am also happy to let go, if its already
> > clear
> > > :)
> > >
> > > Please use the bullet number when you share your feedback so we know
> what
> > > the discussion is about.
> > >
> > > Thanks
> > > Vinoth
> > >
> >
>

Re: [DISCUSS] Simplification of terminologies

[vino yang <ya...@gmail.com>]

Hi Vinoth,

Thanks for bringing these proposals.

+1 on all three. Especially, big +1 on the third renaming proposal.

When I was a newbie. The "COPY_ON_WRITE" term confused me a lot. It easily
mislead users on the "copy" term. And make users compare it with the
`CopyOnWriteArrayList` data structure provided by JDK  and thoughts of the
file systems.

Best,
Vino


Bhavani Sudha <bh...@gmail.com> 于2019年11月12日周二 上午9:05写道：

> +1 on all three rename proposals. I think this would make the concepts
> super easy to follow for new users.
>
> If changing [3] seems to be a stretch, we should definitely do [1] & [2] at
> the least IMO. I will be glad to help out on the renames to whatever extent
> possible should the Hudi community incline to pursue this.
>
> Thanks,
> Sudha
>
>
>
> On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hello all,
> >
> > I wanted to raise an important topic with the community around whether we
> > should rename some of our terminologies in code/docs to be more
> > user-friendly and understandable..
> >
> > Let me also provide some context for each, since I am probably guilty of
> > introducing most of them in the first place :).
> >
> > *1. Rename "views" to "query" : *Instead of saying incremental view or
> > read-optimized view, talk about them as "incremental query" and
> > "read-optimized query". The term "view" is very technical, and what I was
> > trying to convey was that we ingest/store the data once and expose views
> on
> > top. But new users (atleast half dozen of them to me) tend to confuse
> this
> > with views/materialized views found in databases. Almost always we talk
> > about views mostly in terms of expected behavior for a query on the
> view. I
> > am proposing to just call these different query types since its a more
> > universally accepted terminology and IMO clearer.
> >
> > *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> > Read-Optimized view only for MOR storage :* This one is probably the
> > trickiest. Hudi was always designed with MOR in mind, even as we were
> > working on COW storage and consequently we named the pure parquet backed
> > view as Read-Optimized, hoping to name parquet + avro based view as
> > Write-Optimized. However, we opted to name it Realtime to emphasize the
> > data freshness aspect. In retrospect, the views should have not been
> named
> > after their performance characteristics but rather the classes of queries
> > done on them and guarantees for those (point above #1). Moreover, once we
> > have parquet embedded into the log format, then the tradeoffs may not be
> > the same anyways.
> >
> > So combining with the renaming proposed in #1, we would end up with the
> > following..
> >
> > Copy-On-Write :
> > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > [Old]  Incremental View => [New] Incremental Query
> >
> > Merge-On-Read:
> > [Old] Realtime View => [New] Snapshot Query
> > [Old] Incremental View => [New] Incremental Query
> > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is read
> > optimized compared to Snapshot query always, at the cost of staler data)
> >
> > Both changes #1 & #2 could be simpler changes to just code references,
> docs
> > and configs.. we can support both string for sometime and deprecate
> > eventually since queries are stateless.
> >
> > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since the
> > design was very similar to https://en.wikipedia.org/wiki/Copy-on-write
> > filesystems
> > & snapshotting and we once hoped to push some of this logic into the
> > storage itself, all in vain. but the name stuck, even though once we had
> > MERGE_ON_READ the focus was often on merge costs etc, which the name
> > COPY_ON_WRITE does not convey directly. I don't feel very strong about
> this
> > and there is also cost to changing this since its persisted inside
> > hoodie.properties and we will support both strings internally in code for
> > backwards compatibility anyway
> >
> > Naming something is very hard (yes, try :)).I believe these changes will
> > make the project simpler to understand for everyone out there. We also
> have
> > tons of new people here, so I am also happy to let go, if its already
> clear
> > :)
> >
> > Please use the bullet number when you share your feedback so we know what
> > the discussion is about.
> >
> > Thanks
> > Vinoth
> >
>

Re: [DISCUSS] Simplification of terminologies

[Shiyan Xu <xu...@gmail.com>]

[1] +1; "query" indeed sounds better
[2] +1 on the term "snapshot"; so basically we follow the convention that
when we say "snapshot", it means "give me the most up-to-date facts (lowest
data latency) even if it takes some query time"
[3] Though I agree with the renaming, I have a different perspective to
raise on the table types:
MOR is a superset of COW; I suppose a user can theoretically configure a
Hudi streamer to write to a MOR table and make it behave equivalently to a
COW table, am I right? I imagine that involves scheduling compaction/clean
right after write operations and hence making RO view and RT view close to
each other. So what will be the advantage of defining COW tables if MOR can
do everything? Would be happy to get more insights on the benefits of
defining COW over MOR.
So based on COW being an subset and yielded from a special configuration of
MOR, my thoughts are: can we just keep MOR and deprecate COW? In case when
users don't need RT view, can we provide a flag like
"--disable-realtime-view/query" to help achieve the original COW features?
So back to the renaming, if COW can be achieved by changing configs of MOR,
then we could potentially save the hassles of renaming and just deprecate
the type.

On Mon, Nov 11, 2019 at 5:05 PM Bhavani Sudha <bh...@gmail.com>
wrote:

> +1 on all three rename proposals. I think this would make the concepts
> super easy to follow for new users.
>
> If changing [3] seems to be a stretch, we should definitely do [1] & [2] at
> the least IMO. I will be glad to help out on the renames to whatever extent
> possible should the Hudi community incline to pursue this.
>
> Thanks,
> Sudha
>
>
>
> On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org> wrote:
>
> > Hello all,
> >
> > I wanted to raise an important topic with the community around whether we
> > should rename some of our terminologies in code/docs to be more
> > user-friendly and understandable..
> >
> > Let me also provide some context for each, since I am probably guilty of
> > introducing most of them in the first place :).
> >
> > *1. Rename "views" to "query" : *Instead of saying incremental view or
> > read-optimized view, talk about them as "incremental query" and
> > "read-optimized query". The term "view" is very technical, and what I was
> > trying to convey was that we ingest/store the data once and expose views
> on
> > top. But new users (atleast half dozen of them to me) tend to confuse
> this
> > with views/materialized views found in databases. Almost always we talk
> > about views mostly in terms of expected behavior for a query on the
> view. I
> > am proposing to just call these different query types since its a more
> > universally accepted terminology and IMO clearer.
> >
> > *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> > Read-Optimized view only for MOR storage :* This one is probably the
> > trickiest. Hudi was always designed with MOR in mind, even as we were
> > working on COW storage and consequently we named the pure parquet backed
> > view as Read-Optimized, hoping to name parquet + avro based view as
> > Write-Optimized. However, we opted to name it Realtime to emphasize the
> > data freshness aspect. In retrospect, the views should have not been
> named
> > after their performance characteristics but rather the classes of queries
> > done on them and guarantees for those (point above #1). Moreover, once we
> > have parquet embedded into the log format, then the tradeoffs may not be
> > the same anyways.
> >
> > So combining with the renaming proposed in #1, we would end up with the
> > following..
> >
> > Copy-On-Write :
> > [Old]  Read-Optimized View =>  [New] Snapshot Query
> > [Old]  Incremental View => [New] Incremental Query
> >
> > Merge-On-Read:
> > [Old] Realtime View => [New] Snapshot Query
> > [Old] Incremental View => [New] Incremental Query
> > [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is read
> > optimized compared to Snapshot query always, at the cost of staler data)
> >
> > Both changes #1 & #2 could be simpler changes to just code references,
> docs
> > and configs.. we can support both string for sometime and deprecate
> > eventually since queries are stateless.
> >
> > *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since the
> > design was very similar to https://en.wikipedia.org/wiki/Copy-on-write
> > filesystems
> > & snapshotting and we once hoped to push some of this logic into the
> > storage itself, all in vain. but the name stuck, even though once we had
> > MERGE_ON_READ the focus was often on merge costs etc, which the name
> > COPY_ON_WRITE does not convey directly. I don't feel very strong about
> this
> > and there is also cost to changing this since its persisted inside
> > hoodie.properties and we will support both strings internally in code for
> > backwards compatibility anyway
> >
> > Naming something is very hard (yes, try :)).I believe these changes will
> > make the project simpler to understand for everyone out there. We also
> have
> > tons of new people here, so I am also happy to let go, if its already
> clear
> > :)
> >
> > Please use the bullet number when you share your feedback so we know what
> > the discussion is about.
> >
> > Thanks
> > Vinoth
> >
>

Re: [DISCUSS] Simplification of terminologies

[Bhavani Sudha <bh...@gmail.com>]

+1 on all three rename proposals. I think this would make the concepts
super easy to follow for new users.

If changing [3] seems to be a stretch, we should definitely do [1] & [2] at
the least IMO. I will be glad to help out on the renames to whatever extent
possible should the Hudi community incline to pursue this.

Thanks,
Sudha



On Mon, Nov 11, 2019 at 3:46 PM Vinoth Chandar <vi...@apache.org> wrote:

> Hello all,
>
> I wanted to raise an important topic with the community around whether we
> should rename some of our terminologies in code/docs to be more
> user-friendly and understandable..
>
> Let me also provide some context for each, since I am probably guilty of
> introducing most of them in the first place :).
>
> *1. Rename "views" to "query" : *Instead of saying incremental view or
> read-optimized view, talk about them as "incremental query" and
> "read-optimized query". The term "view" is very technical, and what I was
> trying to convey was that we ingest/store the data once and expose views on
> top. But new users (atleast half dozen of them to me) tend to confuse this
> with views/materialized views found in databases. Almost always we talk
> about views mostly in terms of expected behavior for a query on the view. I
> am proposing to just call these different query types since its a more
> universally accepted terminology and IMO clearer.
>
> *2. Rename "Read-Optimized/Realtime" views to Snapshot views + Have
> Read-Optimized view only for MOR storage :* This one is probably the
> trickiest. Hudi was always designed with MOR in mind, even as we were
> working on COW storage and consequently we named the pure parquet backed
> view as Read-Optimized, hoping to name parquet + avro based view as
> Write-Optimized. However, we opted to name it Realtime to emphasize the
> data freshness aspect. In retrospect, the views should have not been named
> after their performance characteristics but rather the classes of queries
> done on them and guarantees for those (point above #1). Moreover, once we
> have parquet embedded into the log format, then the tradeoffs may not be
> the same anyways.
>
> So combining with the renaming proposed in #1, we would end up with the
> following..
>
> Copy-On-Write :
> [Old]  Read-Optimized View =>  [New] Snapshot Query
> [Old]  Incremental View => [New] Incremental Query
>
> Merge-On-Read:
> [Old] Realtime View => [New] Snapshot Query
> [Old] Incremental View => [New] Incremental Query
> [Old] ReadOptimzied View => [New] Read-Optimized Query (since it is read
> optimized compared to Snapshot query always, at the cost of staler data)
>
> Both changes #1 & #2 could be simpler changes to just code references, docs
> and configs.. we can support both string for sometime and deprecate
> eventually since queries are stateless.
>
> *3. Rename COPY_ON_WRITE to MERGE_ON_WRITE :* Name originated since the
> design was very similar to https://en.wikipedia.org/wiki/Copy-on-write
> filesystems
> & snapshotting and we once hoped to push some of this logic into the
> storage itself, all in vain. but the name stuck, even though once we had
> MERGE_ON_READ the focus was often on merge costs etc, which the name
> COPY_ON_WRITE does not convey directly. I don't feel very strong about this
> and there is also cost to changing this since its persisted inside
> hoodie.properties and we will support both strings internally in code for
> backwards compatibility anyway
>
> Naming something is very hard (yes, try :)).I believe these changes will
> make the project simpler to understand for everyone out there. We also have
> tons of new people here, so I am also happy to let go, if its already clear
> :)
>
> Please use the bullet number when you share your feedback so we know what
> the discussion is about.
>
> Thanks
> Vinoth
>