You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Michael Edge <ed...@gmail.com> on 2015/12/01 03:30:10 UTC
Re: UML sequence diagrams on Wiki for explaining read/write path
Thanks for the feedback guys. I've made the updates.
On 1 December 2015 at 00:56, Jack Krupansky <ja...@gmail.com>
wrote:
> Great stuff!
>
> You wrote "When the replica node comes back online the coordinator node
> will send the data to the replica node", which is partially true - if the
> replica comes back online within the timeout window of three hours. So, you
> probably want to say something like:
>
> "If the replica node comes back online within the hinted handoff window the
> coordinator node will send the data to the replica node, otherwise the
> replica node will need to be repaired."
>
> And maybe mention the configuration of the window. Change "The data is
> stored for a default period of 3 hours" to "The data is stored for a
> default period of 3 hours, configurable using the
> max_hint_window_in_ms property
> in cassandra.yaml."
>
> -- Jack Krupansky
>
> On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <ed...@gmail.com>
> wrote:
>
> > Write path docs updated on Wiki - please review diagram/text and let me
> > have your comments (or update text in place).
> >
> > https://wiki.apache.org/cassandra/WritePathForUsers
> >
> > Cheers,
> >
> > Michael
> >
> > On 26 November 2015 at 11:25, Michael Shuler <mi...@pbandjelly.org>
> > wrote:
> >
> > > On 11/25/2015 07:36 PM, Michael Edge wrote:
> > > > I'd like to update the read/write path description on the wiki (see
> > link
> > > > below) by adding a couple of UML sequence diagrams I drew a while
> ago.
> > I
> > > > think they are much better than long textual descriptions for
> > describing
> > > > the order of operations on components. Before publishing them I'd
> > prefer
> > > a
> > > > knowledgeable volunteer to review them to ensure they are accurate.
> > > >
> > > > Let me know if you're interested and I'll send you a copy.
> > > >
> > > > https://wiki.apache.org/cassandra/ArchitectureOverview
> > >
> > > I don't think the mailing list likes attachments, so throw them up on a
> > > web server somewhere and send the list link(s) to your diagrams.
> > > Alternatively, go ahead and post them on the wiki and ask for feedback.
> > >
> > > --
> > > Kind regards,
> > > Michael
> > >
> >
>
Re: UML sequence diagrams on Wiki for explaining read/write path
Posted by Michael Edge <ed...@gmail.com>.
Thanks for reviewing and providing comments. I've created links on the Wiki
front page for the Read/Write path pages.
Details on MV population are probably better left to another page, so I'll
leave those for now.
One thing I'm unclear on in the Read Path is the relationship of Table to
MemTable:
CollationController code shows that multiple MemTables are merged. My
understanding is that only 1 MemTable is active – once it is flushed the
MemTable is no longer used. I’m not sure why we are scanning multiple
MemTables. Is it due to the handover between MemTables at flush time? When
an existing MemTable is about to become non-active and a new one taking
it’s place?
On 3 December 2015 at 00:05, Jack Krupansky <ja...@gmail.com>
wrote:
> Thanks. I took a quick look and it seems fine, but... I don't have the
> depth of expertise in that code to be 100% sure of each detail. Hopefully
> Carl or Jake, et al can review.
>
> One additional point on MV: When a new MV is created for a base table that
> is already populated, that kicks off a backfilling process that will read
> each existing row from the base table and apply the MV update process. I'm
> not sure of the precise details there either (like, which consistency those
> backfill writes use.) And that backfilling process has to occur on each
> node of the cluster (but not each replica.) Again, Carl, Jake, et al need
> to review precise details.
>
> -- Jack Krupansky
>
> On Wed, Dec 2, 2015 at 1:02 AM, Michael Edge <ed...@gmail.com>
> wrote:
>
> > Yes - good point.
> >
> > I've updated the text for the write path page below - could you please
> > review? Once my understanding is correct I'll update the sequence diagram
> > to match.
> >
> > https://wiki.apache.org/cassandra/WritePathForUsers
> >
> > On 1 December 2015 at 22:51, Jack Krupansky <ja...@gmail.com>
> > wrote:
> >
> > > I just remembered... the new Materialized View support in 3.0 - writes
> to
> > > the materialized views get triggered when a write occurs to the base
> > table.
> > > That needs to be in the write path flow/description as well.
> > >
> > > -- Jack Krupansky
> > >
> > > On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge <ed...@gmail.com>
> > > wrote:
> > >
> > > > Thanks for the feedback guys. I've made the updates.
> > > >
> > > > On 1 December 2015 at 00:56, Jack Krupansky <
> jack.krupansky@gmail.com>
> > > > wrote:
> > > >
> > > > > Great stuff!
> > > > >
> > > > > You wrote "When the replica node comes back online the coordinator
> > node
> > > > > will send the data to the replica node", which is partially true -
> if
> > > the
> > > > > replica comes back online within the timeout window of three hours.
> > So,
> > > > you
> > > > > probably want to say something like:
> > > > >
> > > > > "If the replica node comes back online within the hinted handoff
> > window
> > > > the
> > > > > coordinator node will send the data to the replica node, otherwise
> > the
> > > > > replica node will need to be repaired."
> > > > >
> > > > > And maybe mention the configuration of the window. Change "The data
> > is
> > > > > stored for a default period of 3 hours" to "The data is stored for
> a
> > > > > default period of 3 hours, configurable using the
> > > > > max_hint_window_in_ms property
> > > > > in cassandra.yaml."
> > > > >
> > > > > -- Jack Krupansky
> > > > >
> > > > > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <
> > edge.michael@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Write path docs updated on Wiki - please review diagram/text and
> > let
> > > me
> > > > > > have your comments (or update text in place).
> > > > > >
> > > > > > https://wiki.apache.org/cassandra/WritePathForUsers
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Michael
> > > > > >
> > > > > > On 26 November 2015 at 11:25, Michael Shuler <
> > michael@pbandjelly.org
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > On 11/25/2015 07:36 PM, Michael Edge wrote:
> > > > > > > > I'd like to update the read/write path description on the
> wiki
> > > (see
> > > > > > link
> > > > > > > > below) by adding a couple of UML sequence diagrams I drew a
> > while
> > > > > ago.
> > > > > > I
> > > > > > > > think they are much better than long textual descriptions for
> > > > > > describing
> > > > > > > > the order of operations on components. Before publishing them
> > I'd
> > > > > > prefer
> > > > > > > a
> > > > > > > > knowledgeable volunteer to review them to ensure they are
> > > accurate.
> > > > > > > >
> > > > > > > > Let me know if you're interested and I'll send you a copy.
> > > > > > > >
> > > > > > > > https://wiki.apache.org/cassandra/ArchitectureOverview
> > > > > > >
> > > > > > > I don't think the mailing list likes attachments, so throw them
> > up
> > > > on a
> > > > > > > web server somewhere and send the list link(s) to your
> diagrams.
> > > > > > > Alternatively, go ahead and post them on the wiki and ask for
> > > > feedback.
> > > > > > >
> > > > > > > --
> > > > > > > Kind regards,
> > > > > > > Michael
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: UML sequence diagrams on Wiki for explaining read/write path
Posted by Jack Krupansky <ja...@gmail.com>.
Thanks. I took a quick look and it seems fine, but... I don't have the
depth of expertise in that code to be 100% sure of each detail. Hopefully
Carl or Jake, et al can review.
One additional point on MV: When a new MV is created for a base table that
is already populated, that kicks off a backfilling process that will read
each existing row from the base table and apply the MV update process. I'm
not sure of the precise details there either (like, which consistency those
backfill writes use.) And that backfilling process has to occur on each
node of the cluster (but not each replica.) Again, Carl, Jake, et al need
to review precise details.
-- Jack Krupansky
On Wed, Dec 2, 2015 at 1:02 AM, Michael Edge <ed...@gmail.com> wrote:
> Yes - good point.
>
> I've updated the text for the write path page below - could you please
> review? Once my understanding is correct I'll update the sequence diagram
> to match.
>
> https://wiki.apache.org/cassandra/WritePathForUsers
>
> On 1 December 2015 at 22:51, Jack Krupansky <ja...@gmail.com>
> wrote:
>
> > I just remembered... the new Materialized View support in 3.0 - writes to
> > the materialized views get triggered when a write occurs to the base
> table.
> > That needs to be in the write path flow/description as well.
> >
> > -- Jack Krupansky
> >
> > On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge <ed...@gmail.com>
> > wrote:
> >
> > > Thanks for the feedback guys. I've made the updates.
> > >
> > > On 1 December 2015 at 00:56, Jack Krupansky <ja...@gmail.com>
> > > wrote:
> > >
> > > > Great stuff!
> > > >
> > > > You wrote "When the replica node comes back online the coordinator
> node
> > > > will send the data to the replica node", which is partially true - if
> > the
> > > > replica comes back online within the timeout window of three hours.
> So,
> > > you
> > > > probably want to say something like:
> > > >
> > > > "If the replica node comes back online within the hinted handoff
> window
> > > the
> > > > coordinator node will send the data to the replica node, otherwise
> the
> > > > replica node will need to be repaired."
> > > >
> > > > And maybe mention the configuration of the window. Change "The data
> is
> > > > stored for a default period of 3 hours" to "The data is stored for a
> > > > default period of 3 hours, configurable using the
> > > > max_hint_window_in_ms property
> > > > in cassandra.yaml."
> > > >
> > > > -- Jack Krupansky
> > > >
> > > > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <
> edge.michael@gmail.com>
> > > > wrote:
> > > >
> > > > > Write path docs updated on Wiki - please review diagram/text and
> let
> > me
> > > > > have your comments (or update text in place).
> > > > >
> > > > > https://wiki.apache.org/cassandra/WritePathForUsers
> > > > >
> > > > > Cheers,
> > > > >
> > > > > Michael
> > > > >
> > > > > On 26 November 2015 at 11:25, Michael Shuler <
> michael@pbandjelly.org
> > >
> > > > > wrote:
> > > > >
> > > > > > On 11/25/2015 07:36 PM, Michael Edge wrote:
> > > > > > > I'd like to update the read/write path description on the wiki
> > (see
> > > > > link
> > > > > > > below) by adding a couple of UML sequence diagrams I drew a
> while
> > > > ago.
> > > > > I
> > > > > > > think they are much better than long textual descriptions for
> > > > > describing
> > > > > > > the order of operations on components. Before publishing them
> I'd
> > > > > prefer
> > > > > > a
> > > > > > > knowledgeable volunteer to review them to ensure they are
> > accurate.
> > > > > > >
> > > > > > > Let me know if you're interested and I'll send you a copy.
> > > > > > >
> > > > > > > https://wiki.apache.org/cassandra/ArchitectureOverview
> > > > > >
> > > > > > I don't think the mailing list likes attachments, so throw them
> up
> > > on a
> > > > > > web server somewhere and send the list link(s) to your diagrams.
> > > > > > Alternatively, go ahead and post them on the wiki and ask for
> > > feedback.
> > > > > >
> > > > > > --
> > > > > > Kind regards,
> > > > > > Michael
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: UML sequence diagrams on Wiki for explaining read/write path
Posted by Michael Edge <ed...@gmail.com>.
Yes - good point.
I've updated the text for the write path page below - could you please
review? Once my understanding is correct I'll update the sequence diagram
to match.
https://wiki.apache.org/cassandra/WritePathForUsers
On 1 December 2015 at 22:51, Jack Krupansky <ja...@gmail.com>
wrote:
> I just remembered... the new Materialized View support in 3.0 - writes to
> the materialized views get triggered when a write occurs to the base table.
> That needs to be in the write path flow/description as well.
>
> -- Jack Krupansky
>
> On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge <ed...@gmail.com>
> wrote:
>
> > Thanks for the feedback guys. I've made the updates.
> >
> > On 1 December 2015 at 00:56, Jack Krupansky <ja...@gmail.com>
> > wrote:
> >
> > > Great stuff!
> > >
> > > You wrote "When the replica node comes back online the coordinator node
> > > will send the data to the replica node", which is partially true - if
> the
> > > replica comes back online within the timeout window of three hours. So,
> > you
> > > probably want to say something like:
> > >
> > > "If the replica node comes back online within the hinted handoff window
> > the
> > > coordinator node will send the data to the replica node, otherwise the
> > > replica node will need to be repaired."
> > >
> > > And maybe mention the configuration of the window. Change "The data is
> > > stored for a default period of 3 hours" to "The data is stored for a
> > > default period of 3 hours, configurable using the
> > > max_hint_window_in_ms property
> > > in cassandra.yaml."
> > >
> > > -- Jack Krupansky
> > >
> > > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <ed...@gmail.com>
> > > wrote:
> > >
> > > > Write path docs updated on Wiki - please review diagram/text and let
> me
> > > > have your comments (or update text in place).
> > > >
> > > > https://wiki.apache.org/cassandra/WritePathForUsers
> > > >
> > > > Cheers,
> > > >
> > > > Michael
> > > >
> > > > On 26 November 2015 at 11:25, Michael Shuler <michael@pbandjelly.org
> >
> > > > wrote:
> > > >
> > > > > On 11/25/2015 07:36 PM, Michael Edge wrote:
> > > > > > I'd like to update the read/write path description on the wiki
> (see
> > > > link
> > > > > > below) by adding a couple of UML sequence diagrams I drew a while
> > > ago.
> > > > I
> > > > > > think they are much better than long textual descriptions for
> > > > describing
> > > > > > the order of operations on components. Before publishing them I'd
> > > > prefer
> > > > > a
> > > > > > knowledgeable volunteer to review them to ensure they are
> accurate.
> > > > > >
> > > > > > Let me know if you're interested and I'll send you a copy.
> > > > > >
> > > > > > https://wiki.apache.org/cassandra/ArchitectureOverview
> > > > >
> > > > > I don't think the mailing list likes attachments, so throw them up
> > on a
> > > > > web server somewhere and send the list link(s) to your diagrams.
> > > > > Alternatively, go ahead and post them on the wiki and ask for
> > feedback.
> > > > >
> > > > > --
> > > > > Kind regards,
> > > > > Michael
> > > > >
> > > >
> > >
> >
>
Re: UML sequence diagrams on Wiki for explaining read/write path
Posted by Jack Krupansky <ja...@gmail.com>.
I just remembered... the new Materialized View support in 3.0 - writes to
the materialized views get triggered when a write occurs to the base table.
That needs to be in the write path flow/description as well.
-- Jack Krupansky
On Mon, Nov 30, 2015 at 9:30 PM, Michael Edge <ed...@gmail.com>
wrote:
> Thanks for the feedback guys. I've made the updates.
>
> On 1 December 2015 at 00:56, Jack Krupansky <ja...@gmail.com>
> wrote:
>
> > Great stuff!
> >
> > You wrote "When the replica node comes back online the coordinator node
> > will send the data to the replica node", which is partially true - if the
> > replica comes back online within the timeout window of three hours. So,
> you
> > probably want to say something like:
> >
> > "If the replica node comes back online within the hinted handoff window
> the
> > coordinator node will send the data to the replica node, otherwise the
> > replica node will need to be repaired."
> >
> > And maybe mention the configuration of the window. Change "The data is
> > stored for a default period of 3 hours" to "The data is stored for a
> > default period of 3 hours, configurable using the
> > max_hint_window_in_ms property
> > in cassandra.yaml."
> >
> > -- Jack Krupansky
> >
> > On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <ed...@gmail.com>
> > wrote:
> >
> > > Write path docs updated on Wiki - please review diagram/text and let me
> > > have your comments (or update text in place).
> > >
> > > https://wiki.apache.org/cassandra/WritePathForUsers
> > >
> > > Cheers,
> > >
> > > Michael
> > >
> > > On 26 November 2015 at 11:25, Michael Shuler <mi...@pbandjelly.org>
> > > wrote:
> > >
> > > > On 11/25/2015 07:36 PM, Michael Edge wrote:
> > > > > I'd like to update the read/write path description on the wiki (see
> > > link
> > > > > below) by adding a couple of UML sequence diagrams I drew a while
> > ago.
> > > I
> > > > > think they are much better than long textual descriptions for
> > > describing
> > > > > the order of operations on components. Before publishing them I'd
> > > prefer
> > > > a
> > > > > knowledgeable volunteer to review them to ensure they are accurate.
> > > > >
> > > > > Let me know if you're interested and I'll send you a copy.
> > > > >
> > > > > https://wiki.apache.org/cassandra/ArchitectureOverview
> > > >
> > > > I don't think the mailing list likes attachments, so throw them up
> on a
> > > > web server somewhere and send the list link(s) to your diagrams.
> > > > Alternatively, go ahead and post them on the wiki and ask for
> feedback.
> > > >
> > > > --
> > > > Kind regards,
> > > > Michael
> > > >
> > >
> >
>
Re: UML sequence diagrams on Wiki for explaining read/write path
Posted by Michael Edge <ed...@gmail.com>.
Read Path ready for review:
https://wiki.apache.org/cassandra/ReadPathForUsers
On 1 December 2015 at 10:30, Michael Edge <ed...@gmail.com> wrote:
> Thanks for the feedback guys. I've made the updates.
>
> On 1 December 2015 at 00:56, Jack Krupansky <ja...@gmail.com>
> wrote:
>
>> Great stuff!
>>
>> You wrote "When the replica node comes back online the coordinator node
>> will send the data to the replica node", which is partially true - if the
>> replica comes back online within the timeout window of three hours. So,
>> you
>> probably want to say something like:
>>
>> "If the replica node comes back online within the hinted handoff window
>> the
>> coordinator node will send the data to the replica node, otherwise the
>> replica node will need to be repaired."
>>
>> And maybe mention the configuration of the window. Change "The data is
>> stored for a default period of 3 hours" to "The data is stored for a
>> default period of 3 hours, configurable using the
>> max_hint_window_in_ms property
>> in cassandra.yaml."
>>
>> -- Jack Krupansky
>>
>> On Mon, Nov 30, 2015 at 2:24 AM, Michael Edge <ed...@gmail.com>
>> wrote:
>>
>> > Write path docs updated on Wiki - please review diagram/text and let me
>> > have your comments (or update text in place).
>> >
>> > https://wiki.apache.org/cassandra/WritePathForUsers
>> >
>> > Cheers,
>> >
>> > Michael
>> >
>> > On 26 November 2015 at 11:25, Michael Shuler <mi...@pbandjelly.org>
>> > wrote:
>> >
>> > > On 11/25/2015 07:36 PM, Michael Edge wrote:
>> > > > I'd like to update the read/write path description on the wiki (see
>> > link
>> > > > below) by adding a couple of UML sequence diagrams I drew a while
>> ago.
>> > I
>> > > > think they are much better than long textual descriptions for
>> > describing
>> > > > the order of operations on components. Before publishing them I'd
>> > prefer
>> > > a
>> > > > knowledgeable volunteer to review them to ensure they are accurate.
>> > > >
>> > > > Let me know if you're interested and I'll send you a copy.
>> > > >
>> > > > https://wiki.apache.org/cassandra/ArchitectureOverview
>> > >
>> > > I don't think the mailing list likes attachments, so throw them up on
>> a
>> > > web server somewhere and send the list link(s) to your diagrams.
>> > > Alternatively, go ahead and post them on the wiki and ask for
>> feedback.
>> > >
>> > > --
>> > > Kind regards,
>> > > Michael
>> > >
>> >
>>
>
>