You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Anuj Wadehra <an...@yahoo.co.in.INVALID> on 2018/08/17 11:06:39 UTC
Re: URGENT: CASSANDRA-14092 causes Data Loss
Hi,
I think CASSANDRA-14227 is pending for long time now. Though, the data loss issue was addressed in CASSANDRA-14092, Cassandra users are still prohibited to use long TTLs (20+ years) as the maximum expiration timestamp that can be represented by the storage engine is 2038-01-19T03:14:06+00:00 (due to the encoding of localExpirationTime as an int32). As per JIRA comments, the fix seems relatively simple. Considering high impact/returns and relatively less efforts, are there any plans to prioritize this fix for upcoming releases?
Thanks
Anuj
On Saturday, 27 January, 2018, 8:35:20 PM IST, Anuj Wadehra <an...@yahoo.co.in.INVALID> wrote:
Hi Paulo,
Thanks for coming out with the Emergency Hot Fix!!
The patch will help many Cassandra users in saving their precious data.
I think the criticality and urgency of the bug is too high. How can we make sure that maximum Cassandra users are alerted about the silent deletion problem? What are formal ways of working for broadcasting such critical alerts?
I still see that the JIRA is marked as a "Major" defect and not a "Blocker". What worst can happen to a database than irrecoverable silent deletion of successfully inserted data. I hope you understand.
ThanksAnuj
On Fri, 26 Jan 2018 at 18:57, Paulo Motta<pa...@gmail.com> wrote: > I have serious concerns regarding reducing the TTL to 15 yrs.The patch will immediately break all existing applications in Production which are using 15+ yrs TTL.
In order to prevent applications from breaking I will update the patch
to automatically set the maximum TTL to '03:14:08 UTC 19 January 2038'
when it overflows and log a warning as a initial measure. We will
work on extending this limit or lifting this limitation, probably for
the 3.0+ series due to the large scale compatibility changes required
on lower versions, but community patches are always welcome.
Companies that cannot upgrade to a version with the proper fix will
need to workaround this limitation in some other way: do a batch job
to delete old data periodically, perform deletes with timestamps in
the future, etc.
> If its a 32 bit timestamp, can't we just save/read localDeletionTime as unsinged int?
The proper fix will likely be along these lines, but this involve many
changes throughout the codebase where localDeletionTime is consumed
and extensive testing, reviewing, etc, so we're now looking into a
emergency hot fix to prevent silent data loss while the permanent fix is
not in place.
2018-01-26 6:27 GMT-02:00 Anuj Wadehra <an...@yahoo.co.in.invalid>:
> Hi Jeff,
> One correction in my last message: "it may be more feasible to SUPPORT (not extend) the 20 year limit in Cassandra in 2.1/2.2".
> I completely agree that the existing 20 years TTL support is okay for older versions.
>
> If I have understood your last message correctly, upcoming patches are on following lines :
>
> 1. New Patches shall be released for 2.1, 2.2 and 3.x.2. The patches for 2.1 & 2.2 would support the existing 20 year TTL limit and ensure that there is no data loss when 20 year is set as TTL.3. The patches for 2.1 and 2.2 are unlikely to update the sstable format.
> 4. 3.x patches may even remove the 20 year TTL constraint (and extend TTL support beyond 2038).
> I think that the JIRA priority should be increased from "Major" to "Blocker" as the JIRA may cause unexpected data loss. Also, all impacted versions should be included in the JIRA. This will attract the due attention of all Cassandra users.
> ThanksAnuj
> On Friday 26 January 2018, 12:47:18 PM IST, Anuj Wadehra <an...@yahoo.co.in.INVALID> wrote:
>
> Hi Jeff,
>
> Thanks for the prompt action! I agree that patching an application MAY have a shorter life cycle than patching Cassandra in production. But, in the interest of the larger Cassandra user community, we should put our best effort to avoid breaking all the affected applications in production. We should also consider that updating business logic as per the new 15 year TTL constraint may have business implications for many users. I have a limited understanding about the complexity of the code patch, but it may be more feasible to extend the 20 year limit in Cassandra in 2.1/2.2 rather than asking all impacted users to do an immediate business logic adaptation. Moreover, now that we officially support Cassandra 2.1 & 2.2 until 4.0 release and provide critical fixes for 2.1, it becomes even more reasonable to provide this extremely critical patch for 2.1 & 2.2 (unless its absolutely impossible). Still, many users use Cassandra 2.1 and 2.2 in their most critical production systems.
>
> Thanks
> Anuj
>
> On Friday 26 January 2018, 11:06:30 AM IST, Jeff Jirsa <jj...@gmail.com> wrote:
>
> We’ll get patches out. They almost certainly aren’t going to change the sstable format for old versions (unless whoever writes the patch makes a great argument for it), so there’s probably not going to be post-2038 ttl support for 2.1/2.2. For those old versions, we can definitely make it not lose data, but we almost certainly aren’t going to make the ttl go past 2038 in old versions.
>
> More importantly, any company trying to do 20 year ttl’s that’s waiting for a patched version should start by patching their app to not write invalid ttls - your app release cycle is almost certainly faster than db patch / review / test / release / validation, and you can avoid the data loss application side by calculating the ttl explicitly. It’s not the best solution, but it beats doing nothing, and we’re not rushing out a release in less than a day (we haven’t even started a vote, and voting window is 72 hours for members to review and approve or reject the candidate).
>
>
>
> --
> Jeff Jirsa
>
>
>> On Jan 25, 2018, at 9:07 PM, Jeff Jirsa <jj...@gmail.com> wrote:
>>
>> Patches welcome.
>>
>> --
>> Jeff Jirsa
>>
>>
>>> On Jan 25, 2018, at 8:15 PM, Anuj Wadehra <an...@yahoo.co.in.INVALID> wrote:
>>>
>>> Hi Paulo,
>>>
>>> Thanks for looking into the issue on priority. I have serious concerns regarding reducing the TTL to 15 yrs.The patch will immediately break all existing applications in Production which are using 15+ yrs TTL. And then they would be stuck again until all the logic in Production software is modified and the software is upgraded immediately. This may take days. Such heavy downtime is generally not acceptable for any business. Yes, they will not have silent data loss but they would not be able to do any business either. I think the permanent fix must be prioritized and put on extremely fast track. This is a certain Blocker and the impact could be enormous--with and without the 15 year short-term patch.
>>>
>>> And believe me --there are plenty such business use cases where you use very long TTLs such as 20 yrs for compliance and other reasons.
>>>
>>> Thanks
>>> Anuj
>>>
>>> On Friday 26 January 2018, 4:57:13 AM IST, Michael Kjellman <kj...@apple.com> wrote:
>>>
>>> why are people inserting data with a 15+ year TTL? sorta curious about the actual use case for that.
>>>
>>>> On Jan 25, 2018, at 12:36 PM, horschi <ho...@gmail.com> wrote:
>>>>
>>>> The assertion was working fine until yesterday 03:14 UTC.
>>>>
>>>> The long term solution would be to work with a long instead of a int. The
>>>> serialized seems to be a variable-int already, so that should be fine
>>>> already.
>>>>
>>>> If you change the assertion to 15 years, then applications might fail, as
>>>> they might be setting a 15+ year ttl.
>>>>
>>>> regards,
>>>> Christian
>>>>
>>>> On Thu, Jan 25, 2018 at 9:19 PM, Paulo Motta <pa...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for raising this. Agreed this is bad, when I filed
>>>>> CASSANDRA-14092 I thought a write would fail when localDeletionTime
>>>>> overflows (as it is with 2.1), but that doesn't seem to be the case on
>>>>> 3.0+
>>>>>
>>>>> I propose adding the assertion back so writes will fail, and reduce
>>>>> the max TTL to something like 15 years for the time being while we
>>>>> figure a long term solution.
>>>>>
>>>>> 2018-01-25 18:05 GMT-02:00 Jeremiah D Jordan <je...@gmail.com>:
>>>>>> If you aren’t getting an error, then I agree, that is very bad. Looking
>>>>> at the 3.0 code it looks like the assertion checking for overflow was
>>>>> dropped somewhere along the way, I had only been looking into 2.1 where you
>>>>> get an assertion error that fails the query.
>>>>>>
>>>>>> -Jeremiah
>>>>>>
>>>>>>> On Jan 25, 2018, at 2:21 PM, Anuj Wadehra <an...@yahoo.co.in.INVALID>
>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Jeremiah,
>>>>>>> Validation is on TTL value not on (system_time+ TTL). You can test it
>>>>> with below example. Insert is successful, overflow happens silently and
>>>>> data is lost:
>>>>>>> create table test(name text primary key,age int);
>>>>>>> insert into test(name,age) values('test_20yrs',30) USING TTL 630720000;
>>>>>>> select * from test where name='test_20yrs';
>>>>>>>
>>>>>>> name | age
>>>>>>> ------+-----
>>>>>>>
>>>>>>> (0 rows)
>>>>>>>
>>>>>>> insert into test(name,age) values('test_20yr_plus_1',30) USING TTL
>>>>> 630720001;InvalidRequest: Error from server: code=2200 [Invalid query]
>>>>> message="ttl is too large. requested (630720001) maximum (630720000)"
>>>>>>> ThanksAnuj
>>>>>>> On Friday 26 January 2018, 12:11:03 AM IST, J. D. Jordan <
>>>>> jeremiah.jordan@gmail.com> wrote:
>>>>>>>
>>>>>>> Where is the dataloss? Does the INSERT operation return successfully
>>>>> to the client in this case? From reading the linked issues it sounds like
>>>>> you get an error client side.
>>>>>>>
>>>>>>> -Jeremiah
>>>>>>>
>>>>>>>> On Jan 25, 2018, at 1:24 PM, Anuj Wadehra <an...@yahoo.co.in.INVALID>
>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> For all those people who use MAX TTL=20 years for inserting/updating
>>>>> data in production, https://issues.apache.org/jira/browse/CASSANDRA-14092
>>>>> can silently cause irrecoverable Data Loss. This seems like a certain TOP
>>>>> MOST BLOCKER to me. I think the category of the JIRA must be raised to
>>>>> BLOCKER from Major. Unfortunately, the JIRA is still "Unassigned" and no
>>>>> one seems to be actively working on it. Just like any other critical
>>>>> vulnerability, this vulnerability demands immediate attention from some
>>>>> very experienced folks to bring out an Urgent Fast Track Patch for all
>>>>> currently Supported Cassandra versions 2.1,2.2 and 3.x. As per my
>>>>> understanding of the JIRA comments, the changes may not be that trivial for
>>>>> older releases. So, community support on the patch is very much appreciated.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Anuj
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>>
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
>>>>>
>>>>>
>>>
>>> Т���������������������������������������������������������������������ХF�V�7V'67&�&R�R���âFWb�V�7V'67&�&T676�G&�6�R��&pФf�"FF�F����6����G2�R���âFWbֆV�676�G&�6�R��&pР�
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: URGENT: CASSANDRA-14092 causes Data Loss
Posted by kurt greaves <ku...@instaclustr.com>.
I definitely think we should include it in 4.0. TBH I think it's reasonable
for it to get done after the feature freeze seeing as it is a bug.
On 17 August 2018 at 21:06, Anuj Wadehra <an...@yahoo.co.in.invalid>
wrote:
> Hi,
>
> I think CASSANDRA-14227 is pending for long time now. Though, the data
> loss issue was addressed in CASSANDRA-14092, Cassandra users are still
> prohibited to use long TTLs (20+ years) as the maximum expiration timestamp
> that can be represented by the storage engine is 2038-01-19T03:14:06+00:00
> (due to the encoding of localExpirationTime as an int32). As per JIRA
> comments, the fix seems relatively simple. Considering high impact/returns
> and relatively less efforts, are there any plans to prioritize this fix for
> upcoming releases?
>
> Thanks
> Anuj
>
>
>
>
> On Saturday, 27 January, 2018, 8:35:20 PM IST, Anuj Wadehra <
> anujw_2003@yahoo.co.in.INVALID> wrote:
>
>
>
>
>
> Hi Paulo,
>
> Thanks for coming out with the Emergency Hot Fix!!
> The patch will help many Cassandra users in saving their precious data.
> I think the criticality and urgency of the bug is too high. How can we
> make sure that maximum Cassandra users are alerted about the silent
> deletion problem? What are formal ways of working for broadcasting such
> critical alerts?
> I still see that the JIRA is marked as a "Major" defect and not a
> "Blocker". What worst can happen to a database than irrecoverable silent
> deletion of successfully inserted data. I hope you understand.
>
>
>
> ThanksAnuj
>
>
>
>
> On Fri, 26 Jan 2018 at 18:57, Paulo Motta<pa...@gmail.com>
> wrote: > I have serious concerns regarding reducing the TTL to 15 yrs.The
> patch will immediately break all existing applications in Production which
> are using 15+ yrs TTL.
>
> In order to prevent applications from breaking I will update the patch
> to automatically set the maximum TTL to '03:14:08 UTC 19 January 2038'
> when it overflows and log a warning as a initial measure. We will
> work on extending this limit or lifting this limitation, probably for
> the 3.0+ series due to the large scale compatibility changes required
> on lower versions, but community patches are always welcome.
>
> Companies that cannot upgrade to a version with the proper fix will
> need to workaround this limitation in some other way: do a batch job
> to delete old data periodically, perform deletes with timestamps in
> the future, etc.
>
> > If its a 32 bit timestamp, can't we just save/read localDeletionTime as
> unsinged int?
>
> The proper fix will likely be along these lines, but this involve many
> changes throughout the codebase where localDeletionTime is consumed
> and extensive testing, reviewing, etc, so we're now looking into a
> emergency hot fix to prevent silent data loss while the permanent fix is
> not in place.
>
> 2018-01-26 6:27 GMT-02:00 Anuj Wadehra <an...@yahoo.co.in.invalid>:
> > Hi Jeff,
> > One correction in my last message: "it may be more feasible to SUPPORT
> (not extend) the 20 year limit in Cassandra in 2.1/2.2".
> > I completely agree that the existing 20 years TTL support is okay for
> older versions.
> >
> > If I have understood your last message correctly, upcoming patches are
> on following lines :
> >
> > 1. New Patches shall be released for 2.1, 2.2 and 3.x.2. The patches for
> 2.1 & 2.2 would support the existing 20 year TTL limit and ensure that
> there is no data loss when 20 year is set as TTL.3. The patches for 2.1 and
> 2.2 are unlikely to update the sstable format.
> > 4. 3.x patches may even remove the 20 year TTL constraint (and extend
> TTL support beyond 2038).
> > I think that the JIRA priority should be increased from "Major" to
> "Blocker" as the JIRA may cause unexpected data loss. Also, all impacted
> versions should be included in the JIRA. This will attract the due
> attention of all Cassandra users.
> > ThanksAnuj
> > On Friday 26 January 2018, 12:47:18 PM IST, Anuj Wadehra <
> anujw_2003@yahoo.co.in.INVALID> wrote:
> >
> > Hi Jeff,
> >
> > Thanks for the prompt action! I agree that patching an application MAY
> have a shorter life cycle than patching Cassandra in production. But, in
> the interest of the larger Cassandra user community, we should put our best
> effort to avoid breaking all the affected applications in production. We
> should also consider that updating business logic as per the new 15 year
> TTL constraint may have business implications for many users. I have a
> limited understanding about the complexity of the code patch, but it may be
> more feasible to extend the 20 year limit in Cassandra in 2.1/2.2 rather
> than asking all impacted users to do an immediate business logic
> adaptation. Moreover, now that we officially support Cassandra 2.1 & 2.2
> until 4.0 release and provide critical fixes for 2.1, it becomes even more
> reasonable to provide this extremely critical patch for 2.1 & 2.2 (unless
> its absolutely impossible). Still, many users use Cassandra 2.1 and 2.2 in
> their most critical production systems.
> >
> > Thanks
> > Anuj
> >
> > On Friday 26 January 2018, 11:06:30 AM IST, Jeff Jirsa <
> jjirsa@gmail.com> wrote:
> >
> > We’ll get patches out. They almost certainly aren’t going to change the
> sstable format for old versions (unless whoever writes the patch makes a
> great argument for it), so there’s probably not going to be post-2038 ttl
> support for 2.1/2.2. For those old versions, we can definitely make it not
> lose data, but we almost certainly aren’t going to make the ttl go past
> 2038 in old versions.
> >
> > More importantly, any company trying to do 20 year ttl’s that’s waiting
> for a patched version should start by patching their app to not write
> invalid ttls - your app release cycle is almost certainly faster than db
> patch / review / test / release / validation, and you can avoid the data
> loss application side by calculating the ttl explicitly. It’s not the best
> solution, but it beats doing nothing, and we’re not rushing out a release
> in less than a day (we haven’t even started a vote, and voting window is 72
> hours for members to review and approve or reject the candidate).
> >
> >
> >
> > --
> > Jeff Jirsa
> >
> >
> >> On Jan 25, 2018, at 9:07 PM, Jeff Jirsa <jj...@gmail.com> wrote:
> >>
> >> Patches welcome.
> >>
> >> --
> >> Jeff Jirsa
> >>
> >>
> >>> On Jan 25, 2018, at 8:15 PM, Anuj Wadehra <an...@yahoo.co.in.INVALID>
> wrote:
> >>>
> >>> Hi Paulo,
> >>>
> >>> Thanks for looking into the issue on priority. I have serious concerns
> regarding reducing the TTL to 15 yrs.The patch will immediately break all
> existing applications in Production which are using 15+ yrs TTL. And then
> they would be stuck again until all the logic in Production software is
> modified and the software is upgraded immediately. This may take days. Such
> heavy downtime is generally not acceptable for any business. Yes, they will
> not have silent data loss but they would not be able to do any business
> either. I think the permanent fix must be prioritized and put on extremely
> fast track. This is a certain Blocker and the impact could be
> enormous--with and without the 15 year short-term patch.
> >>>
> >>> And believe me --there are plenty such business use cases where you
> use very long TTLs such as 20 yrs for compliance and other reasons.
> >>>
> >>> Thanks
> >>> Anuj
> >>>
> >>> On Friday 26 January 2018, 4:57:13 AM IST, Michael Kjellman <
> kjellman@apple.com> wrote:
> >>>
> >>> why are people inserting data with a 15+ year TTL? sorta curious about
> the actual use case for that.
> >>>
> >>>> On Jan 25, 2018, at 12:36 PM, horschi <ho...@gmail.com> wrote:
> >>>>
> >>>> The assertion was working fine until yesterday 03:14 UTC.
> >>>>
> >>>> The long term solution would be to work with a long instead of a int.
> The
> >>>> serialized seems to be a variable-int already, so that should be fine
> >>>> already.
> >>>>
> >>>> If you change the assertion to 15 years, then applications might
> fail, as
> >>>> they might be setting a 15+ year ttl.
> >>>>
> >>>> regards,
> >>>> Christian
> >>>>
> >>>> On Thu, Jan 25, 2018 at 9:19 PM, Paulo Motta <
> pauloricardomg@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Thanks for raising this. Agreed this is bad, when I filed
> >>>>> CASSANDRA-14092 I thought a write would fail when localDeletionTime
> >>>>> overflows (as it is with 2.1), but that doesn't seem to be the case
> on
> >>>>> 3.0+
> >>>>>
> >>>>> I propose adding the assertion back so writes will fail, and reduce
> >>>>> the max TTL to something like 15 years for the time being while we
> >>>>> figure a long term solution.
> >>>>>
> >>>>> 2018-01-25 18:05 GMT-02:00 Jeremiah D Jordan <
> jeremiah.jordan@gmail.com>:
> >>>>>> If you aren’t getting an error, then I agree, that is very bad.
> Looking
> >>>>> at the 3.0 code it looks like the assertion checking for overflow was
> >>>>> dropped somewhere along the way, I had only been looking into 2.1
> where you
> >>>>> get an assertion error that fails the query.
> >>>>>>
> >>>>>> -Jeremiah
> >>>>>>
> >>>>>>> On Jan 25, 2018, at 2:21 PM, Anuj Wadehra <anujw_2003@yahoo.co.in.
> INVALID>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> Hi Jeremiah,
> >>>>>>> Validation is on TTL value not on (system_time+ TTL). You can test
> it
> >>>>> with below example. Insert is successful, overflow happens silently
> and
> >>>>> data is lost:
> >>>>>>> create table test(name text primary key,age int);
> >>>>>>> insert into test(name,age) values('test_20yrs',30) USING TTL
> 630720000;
> >>>>>>> select * from test where name='test_20yrs';
> >>>>>>>
> >>>>>>> name | age
> >>>>>>> ------+-----
> >>>>>>>
> >>>>>>> (0 rows)
> >>>>>>>
> >>>>>>> insert into test(name,age) values('test_20yr_plus_1',30) USING TTL
> >>>>> 630720001;InvalidRequest: Error from server: code=2200 [Invalid
> query]
> >>>>> message="ttl is too large. requested (630720001) maximum (630720000)"
> >>>>>>> ThanksAnuj
> >>>>>>> On Friday 26 January 2018, 12:11:03 AM IST, J. D. Jordan <
> >>>>> jeremiah.jordan@gmail.com> wrote:
> >>>>>>>
> >>>>>>> Where is the dataloss? Does the INSERT operation return
> successfully
> >>>>> to the client in this case? From reading the linked issues it
> sounds like
> >>>>> you get an error client side.
> >>>>>>>
> >>>>>>> -Jeremiah
> >>>>>>>
> >>>>>>>> On Jan 25, 2018, at 1:24 PM, Anuj Wadehra <anujw_2003@yahoo.co.in
> .INVALID>
> >>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> For all those people who use MAX TTL=20 years for
> inserting/updating
> >>>>> data in production, https://issues.apache.org/
> jira/browse/CASSANDRA-14092
> >>>>> can silently cause irrecoverable Data Loss. This seems like a
> certain TOP
> >>>>> MOST BLOCKER to me. I think the category of the JIRA must be raised
> to
> >>>>> BLOCKER from Major. Unfortunately, the JIRA is still "Unassigned"
> and no
> >>>>> one seems to be actively working on it. Just like any other critical
> >>>>> vulnerability, this vulnerability demands immediate attention from
> some
> >>>>> very experienced folks to bring out an Urgent Fast Track Patch for
> all
> >>>>> currently Supported Cassandra versions 2.1,2.2 and 3.x. As per my
> >>>>> understanding of the JIRA comments, the changes may not be that
> trivial for
> >>>>> older releases. So, community support on the patch is very much
> appreciated.
> >>>>>>>>
> >>>>>>>> Thanks
> >>>>>>>> Anuj
> >>>>>>>
> >>>>>>> ------------------------------------------------------------
> ---------
> >>>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>
> >>>>>>
> >>>>>> ------------------------------------------------------------
> ---------
> >>>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>>
> >>>>>
> >>>>> ------------------------------------------------------------
> ---------
> >>>>> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-help@cassandra.apache.org
> >>>>>
> >>>>>
> >>>
> >>> �����������������������������������������������������������
> ����������ХF�V�7V'67&�&R�R���âFWb�V�7V'67&�&T676�G&�6�R��&
> pФf�"FF�F����6����G2�R���âFWbֆV�676�G&�6�R��&pР�
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
>