Re: [Design Discussion] Transaction manager, time travel and segment interface refactoring

[Venkata Gollamudi <g....@gmail.com>]

+1

Valuable feature for data consistency across tables, mvs, indexes.

On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <ra...@gmail.com>
wrote:

> +1
>
> Much needed feature and interface refactoring. Thanks for working on it.
>
> Regards,
> Ravindra.
>
> On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <aj...@gmail.com>
> wrote:
>
> > Hi All,
> > In this thread, I am continuing the below discussion along with the
> > Transaction Manager and Time Travel feature design.
> >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> >
> > The goal of this requirement is as follows.
> >
> >    1.
> >
> >    Implement a “Transaction Manager” with optimistic concurrency to
> provide
> >    within a table transaction/versioning. (interfaces should also be
> >    flexible enough to support across table transactions)
> >    2.
> >
> >    Support time travel in carbonData.
> >    3.
> >
> >    Decouple and clean up segment interfaces. (which should also help in
> >    supporting segment concepts to other open formats under carbonData
> > metadata
> >    service)
> >
> >
> > The design document is attached in JIRA.
> > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > GoogleDrive link:
> >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> >
> > Please have a look. suggestions are welcome.
> > I have mentioned some TODO in the document, I will be updating it in the
> V2
> > version soon.
> > Implementation will be done by adding subtasks under the same JIRA.
> >
> > Thanks,
> > Ajantha
> >
> --
> Thanks & Regards,
> Ravi
>

Re: [Design Discussion] Transaction manager, time travel and segment interface refactoring

[Kumar Vishal <ku...@gmail.com>]

+1

Small Modification: In current design only the table status file is
getting updated. In this new design tx<Id>_tablestatus[InProgress->Success]
and <db>.<tbl>tx files are getting updated.
I think we can write tx_<Id>_InProgress and tx_<id>_Success to a new file.
and After successful load we can delete the InProgress table status file.

-Regards
Kumar Vishal

On Sun, 2 May 2021 at 10:07 PM, Venkata Gollamudi <g....@gmail.com>
wrote:

> +1
>
> Valuable feature for data consistency across tables, mvs, indexes.
>
> On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <ra...@gmail.com>
> wrote:
>
> > +1
> >
> > Much needed feature and interface refactoring. Thanks for working on it.
> >
> > Regards,
> > Ravindra.
> >
> > On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <aj...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > > In this thread, I am continuing the below discussion along with the
> > > Transaction Manager and Time Travel feature design.
> > >
> > >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> > >
> > > The goal of this requirement is as follows.
> > >
> > >    1.
> > >
> > >    Implement a “Transaction Manager” with optimistic concurrency to
> > provide
> > >    within a table transaction/versioning. (interfaces should also be
> > >    flexible enough to support across table transactions)
> > >    2.
> > >
> > >    Support time travel in carbonData.
> > >    3.
> > >
> > >    Decouple and clean up segment interfaces. (which should also help in
> > >    supporting segment concepts to other open formats under carbonData
> > > metadata
> > >    service)
> > >
> > >
> > > The design document is attached in JIRA.
> > > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > > GoogleDrive link:
> > >
> > >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> > >
> > > Please have a look. suggestions are welcome.
> > > I have mentioned some TODO in the document, I will be updating it in
> the
> > V2
> > > version soon.
> > > Implementation will be done by adding subtasks under the same JIRA.
> > >
> > > Thanks,
> > > Ajantha
> > >
> > --
> > Thanks & Regards,
> > Ravi
> >
>