You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by Ajantha Bhat <aj...@gmail.com> on 2021/04/22 06:36:11 UTC
[Design Discussion] Transaction manager, time travel and segment
interface refactoring
Hi All,
In this thread, I am continuing the below discussion along with the
Transaction Manager and Time Travel feature design.
http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
The goal of this requirement is as follows.
1.
Implement a “Transaction Manager” with optimistic concurrency to provide
within a table transaction/versioning. (interfaces should also be
flexible enough to support across table transactions)
2.
Support time travel in carbonData.
3.
Decouple and clean up segment interfaces. (which should also help in
supporting segment concepts to other open formats under carbonData metadata
service)
The design document is attached in JIRA.
JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
GoogleDrive link:
https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
Please have a look. suggestions are welcome.
I have mentioned some TODO in the document, I will be updating it in the V2
version soon.
Implementation will be done by adding subtasks under the same JIRA.
Thanks,
Ajantha
Re: [Design Discussion] Transaction manager, time travel and segment
interface refactoring
Posted by Kumar Vishal <ku...@gmail.com>.
+1
Small Modification: In current design only the table status file is
getting updated. In this new design tx<Id>_tablestatus[InProgress->Success]
and <db>.<tbl>tx files are getting updated.
I think we can write tx_<Id>_InProgress and tx_<id>_Success to a new file.
and After successful load we can delete the InProgress table status file.
-Regards
Kumar Vishal
On Sun, 2 May 2021 at 10:07 PM, Venkata Gollamudi <g....@gmail.com>
wrote:
> +1
>
> Valuable feature for data consistency across tables, mvs, indexes.
>
> On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <ra...@gmail.com>
> wrote:
>
> > +1
> >
> > Much needed feature and interface refactoring. Thanks for working on it.
> >
> > Regards,
> > Ravindra.
> >
> > On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <aj...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > > In this thread, I am continuing the below discussion along with the
> > > Transaction Manager and Time Travel feature design.
> > >
> > >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> > >
> > > The goal of this requirement is as follows.
> > >
> > > 1.
> > >
> > > Implement a “Transaction Manager” with optimistic concurrency to
> > provide
> > > within a table transaction/versioning. (interfaces should also be
> > > flexible enough to support across table transactions)
> > > 2.
> > >
> > > Support time travel in carbonData.
> > > 3.
> > >
> > > Decouple and clean up segment interfaces. (which should also help in
> > > supporting segment concepts to other open formats under carbonData
> > > metadata
> > > service)
> > >
> > >
> > > The design document is attached in JIRA.
> > > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > > GoogleDrive link:
> > >
> > >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> > >
> > > Please have a look. suggestions are welcome.
> > > I have mentioned some TODO in the document, I will be updating it in
> the
> > V2
> > > version soon.
> > > Implementation will be done by adding subtasks under the same JIRA.
> > >
> > > Thanks,
> > > Ajantha
> > >
> > --
> > Thanks & Regards,
> > Ravi
> >
>
Re: [Design Discussion] Transaction manager, time travel and segment
interface refactoring
Posted by Venkata Gollamudi <g....@gmail.com>.
+1
Valuable feature for data consistency across tables, mvs, indexes.
On Wed, Apr 28, 2021 at 7:02 PM Ravindra Pesala <ra...@gmail.com>
wrote:
> +1
>
> Much needed feature and interface refactoring. Thanks for working on it.
>
> Regards,
> Ravindra.
>
> On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <aj...@gmail.com>
> wrote:
>
> > Hi All,
> > In this thread, I am continuing the below discussion along with the
> > Transaction Manager and Time Travel feature design.
> >
> >
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
> >
> > The goal of this requirement is as follows.
> >
> > 1.
> >
> > Implement a “Transaction Manager” with optimistic concurrency to
> provide
> > within a table transaction/versioning. (interfaces should also be
> > flexible enough to support across table transactions)
> > 2.
> >
> > Support time travel in carbonData.
> > 3.
> >
> > Decouple and clean up segment interfaces. (which should also help in
> > supporting segment concepts to other open formats under carbonData
> > metadata
> > service)
> >
> >
> > The design document is attached in JIRA.
> > JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> > GoogleDrive link:
> >
> >
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
> >
> > Please have a look. suggestions are welcome.
> > I have mentioned some TODO in the document, I will be updating it in the
> V2
> > version soon.
> > Implementation will be done by adding subtasks under the same JIRA.
> >
> > Thanks,
> > Ajantha
> >
> --
> Thanks & Regards,
> Ravi
>
Re: [Design Discussion] Transaction manager, time travel and segment
interface refactoring
Posted by Ravindra Pesala <ra...@gmail.com>.
+1
Much needed feature and interface refactoring. Thanks for working on it.
Regards,
Ravindra.
On Thu, 22 Apr 2021 at 2:36 PM, Ajantha Bhat <aj...@gmail.com> wrote:
> Hi All,
> In this thread, I am continuing the below discussion along with the
> Transaction Manager and Time Travel feature design.
>
> http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/Discussion-Taking-the-inputs-for-Segment-Interface-Refactoring-td101950.html
>
> The goal of this requirement is as follows.
>
> 1.
>
> Implement a “Transaction Manager” with optimistic concurrency to provide
> within a table transaction/versioning. (interfaces should also be
> flexible enough to support across table transactions)
> 2.
>
> Support time travel in carbonData.
> 3.
>
> Decouple and clean up segment interfaces. (which should also help in
> supporting segment concepts to other open formats under carbonData
> metadata
> service)
>
>
> The design document is attached in JIRA.
> JIRA link: https://issues.apache.org/jira/browse/CARBONDATA-4171
> GoogleDrive link:
>
> https://docs.google.com/document/d/1FsVsXjj5QCuFDrzrayN4Qo0LqWc0Kcijc_jL7pCzfXo/edit?usp=sharing
>
> Please have a look. suggestions are welcome.
> I have mentioned some TODO in the document, I will be updating it in the V2
> version soon.
> Implementation will be done by adding subtasks under the same JIRA.
>
> Thanks,
> Ajantha
>
--
Thanks & Regards,
Ravi