You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by ENES ATABERK <en...@turkcell.com.tr> on 2024/01/18 05:21:40 UTC
system_schema.tables id and table uuid on disk mismatch
Hi all,
we have detected that table-uuid in linux file directory is different from system_schema.tables id.
I have executed nodetool describe cluster and see only one schema version in the cluster.
How we can fix this issue do anyone has any idea? Restarting the nodes only create a new empty directory with name system_schema.tables id directory but in this case i have two directories old one has sstables with incorrect uuid new one has correct uuid but empty.
thanks in advance
[https://s.turkcell.com.tr/SiteAssets/Genel/mail/mail-imza.jpg] <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
TURKCELL bu mesajin icerdigi bilgilerin dogrulugu veya eksiksiz oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
________________________________
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted.
TURKCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of TURKCELL.
This e-mail has been scanned for all known computer viruses.
Re: system_schema.tables id and table uuid on disk mismatch
Posted by Štefan Miklošovič <st...@gmail.com>.
Hi,
what Cassandra version are you on? I do not see this behaviour in 5.0-beta.
When I create a table, its id will match the id part of table directory on
the disk.
Regards
On Thu, Jan 18, 2024 at 8:49 AM ENES ATABERK <en...@turkcell.com.tr>
wrote:
> Hi all,
>
> we have detected that table-uuid in linux file directory is different from
> system_schema.tables id.
>
> I have executed nodetool describe cluster and see only one schema version
> in the cluster.
>
> How we can fix this issue do anyone has any idea? Restarting the nodes
> only create a new empty directory with name system_schema.tables id
> directory but in this case i have two directories old one has sstables with
> incorrect uuid new one has correct uuid but empty.
>
> thanks in advance
>
>
>
>
> <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
>
> Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi
> tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi
> icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin
> icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz
> kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
>
> TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu
> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne
> sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve
> saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen
> kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
>
> Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
> ------------------------------
>
> This e-mail and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they are
> addressed. If you are not the intended recipient you are hereby notified
> that any dissemination, forwarding, copying or use of any of the
> information is strictly prohibited, and the e-mail should immediately be
> deleted.
>
> TURKCELL makes no warranty as to the accuracy or completeness of any
> information contained in this message and hereby excludes any liability of
> any kind for the information contained therein or for the information
> transmission, reception, storage or use of such in any way whatsoever. The
> opinions expressed in this message belong to sender alone and may not
> necessarily reflect the opinions of TURKCELL.
>
> This e-mail has been scanned for all known computer viruses.
>
Re: COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Without knowing the cause of the issue, it's hard to tell what are the
correct steps to recover from it. I would recommend you have a look at
the logs and figure out what was the cause of the issue, and then make a
recovery plan and also put preventive measure in place to stop it from
happening again.
Also, I'm not comfortable with the idea of manually creating data
directories for Cassandra unless I know the system well, as this may
lead to filesystem ownership and permission issues. The involvement of
security tools like AppArmor or SELinux may make it even more complicated.
On 18/01/2024 15:46, ENES ATABERK wrote:
>
> ok thank you!
>
> What do you think about the following approach:
>
> 1. creating empty correct table id directories in linux filesystem
> with respect to the system_schema.tables id column
> 2. importing data with nodetool import from incorrect directory
> 3. removing the incorrect directory afterwards
>
> ------------------------------------------------------------------------
> *From:* Bowen Song via user <us...@cassandra.apache.org>
> *Sent:* Thursday, January 18, 2024 5:34:57 PM
> *To:* user@cassandra.apache.org
> *Cc:* Bowen Song
> *Subject:* COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re:
> system_schema.tables id and table uuid on disk mismatch
>
> I know dropping a table and then creating a new table with the same
> name can lead to that result, which is expected. If that wasn't what
> happened, it may be a bug in Cassandra. If you can reproduce the
> behaviour, you should raise a Jira ticket for it.
>
>
> On 18/01/2024 14:44, ENES ATABERK wrote:
>>
>> It has same mismatch id in all nodes not just one node.
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Bowen Song via user <us...@cassandra.apache.org>
>> *Sent:* Thursday, January 18, 2024 3:18:11 PM
>> *To:* user@cassandra.apache.org
>> *Cc:* Bowen Song
>> *Subject:* COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and
>> table uuid on disk mismatch
>>
>> Was the table ID mismatching only on one node or all nodes?
>> Mismatching on one node is usually the result of a racing condition,
>> but on all nodes isn't. The solution I mentioned earlier only applies
>> to the one node situation.
>>
>>
>> On 18/01/2024 13:14, ENES ATABERK wrote:
>>>
>>> Hi all,
>>>
>>> Thanks for your responses.
>>>
>>> The version is Cassandra 4.1.3
>>>
>>> After I restarted all the nodes one-by-one cassandra created
>>> corrected-id folder and keep the incorrect one as you said.
>>>
>>> But then I cannot see the data from cqlsh it gives me no result.
>>> After i have imported the data from incorrect-id-folder i see the data
>>>
>>>
>>> nodetool import keyspace_name table_name
>>> /full_path_of_old(incorrect)_folder.
>>>
>>>
>>> now my questions are like:
>>>
>>> First question; before i have restarted the nodes how can i search
>>> the data although there is a mismatch between system_schema.tables
>>> and actual directories in all nodes.
>>>
>>>
>>> Second one; is nodetool import a safe way to load data from
>>> incorrect folder on a write heavy system. Because I cannot be sure
>>> if I miss any data during the import operation. Or do i need to run
>>> a repair for those tables instead of import? In my opinion that
>>> may not work because i cannot see any data before nodetool import.
>>>
>>>
>>> Thanks again.
>>>
>>>
>>> ------------------------------------------------------------------------
>>> *From:* Bowen Song via user <us...@cassandra.apache.org>
>>> *Sent:* Thursday, January 18, 2024 1:17:11 PM
>>> *To:* user@cassandra.apache.org
>>> *Cc:* Bowen Song
>>> *Subject:* COMMERCIAL:Re: system_schema.tables id and table uuid on
>>> disk mismatch
>>>
>>> It sounds like you have done some concurrent table creation/deletion
>>> in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients),
>>> which resulted in this mismatch. After you restarted the node,
>>> Cassandra corrected it by discarding the old table ID and any data
>>> associated with it. This is the expected behaviour. This issue has
>>> already been fixed, and you can safely delete the data directory
>>> with the incorrect table ID as it is no longer used by Cassandra.
>>> You should now run a full repair on this node to ensure it has all
>>> the data it owns. If you are /absolutely/ certain that the table
>>> with different IDs have identical schema, and the gc_grace_seconds
>>> hasn't past, you may move the data from the wrong data directory to
>>> the correct data directory, and then restart the node or run
>>> "nodetool refresh <keyspace> <table>" on the node before running the
>>> full repair, this may save you some streaming time. However, if the
>>> table schema is different, this may cause a havoc.
>>>
>>>
>>> On 18/01/2024 05:21, ENES ATABERK wrote:
>>>>
>>>> Hi all,
>>>>
>>>> we have detected that table-uuid in linux file directory is
>>>> different from system_schema.tables id.
>>>>
>>>> I have executed nodetool describe cluster and see only one schema
>>>> version in the cluster.
>>>>
>>>> How we can fix this issue do anyone has any idea? Restarting the
>>>> nodes only create a new empty directory with
>>>> name system_schema.tables id directory but in this case i have two
>>>> directories old one has sstables with incorrect uuid new one has
>>>> correct uuid but empty.
>>>>
>>>> thanks in advance
>>>>
>>>>
>>>>
>>>>
>>>> <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
>>>>
>>>> Bu elektronik posta ve onunla iletilen butun dosyalar sadece
>>>> gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel
>>>> kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz
>>>> bu elektronik postanin icerigini aciklamaniz, kopyalamaniz,
>>>> yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik
>>>> postayi derhal silmeniz gerekmektedir.
>>>>
>>>> TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz
>>>> oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu
>>>> bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden,
>>>> alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki
>>>> gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini
>>>> yansitmayabilir
>>>>
>>>> Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>> This e-mail and any files transmitted with it are confidential and
>>>> intended solely for the use of the individual or entity to whom
>>>> they are addressed. If you are not the intended recipient you are
>>>> hereby notified that any dissemination, forwarding, copying or use
>>>> of any of the information is strictly prohibited, and the e-mail
>>>> should immediately be deleted.
>>>>
>>>> TURKCELL makes no warranty as to the accuracy or completeness of
>>>> any information contained in this message and hereby excludes any
>>>> liability of any kind for the information contained therein or for
>>>> the information transmission, reception, storage or use of such in
>>>> any way whatsoever. The opinions expressed in this message belong
>>>> to sender alone and may not necessarily reflect the opinions of
>>>> TURKCELL.
>>>>
>>>> This e-mail has been scanned for all known computer viruses.
>>>>
>>>
>>
>
Re: COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by ENES ATABERK <en...@turkcell.com.tr>.
ok thank you!
What do you think about the following approach:
1. creating empty correct table id directories in linux filesystem with respect to the system_schema.tables id column
2. importing data with nodetool import from incorrect directory
3. removing the incorrect directory afterwards
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 5:34:57 PM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: COMMERCIAL:Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
I know dropping a table and then creating a new table with the same name can lead to that result, which is expected. If that wasn't what happened, it may be a bug in Cassandra. If you can reproduce the behaviour, you should raise a Jira ticket for it.
On 18/01/2024 14:44, ENES ATABERK wrote:
It has same mismatch id in all nodes not just one node.
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 3:18:11 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Cc: Bowen Song
Subject: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Was the table ID mismatching only on one node or all nodes? Mismatching on one node is usually the result of a racing condition, but on all nodes isn't. The solution I mentioned earlier only applies to the one node situation.
On 18/01/2024 13:14, ENES ATABERK wrote:
Hi all,
Thanks for your responses.
The version is Cassandra 4.1.3
After I restarted all the nodes one-by-one cassandra created corrected-id folder and keep the incorrect one as you said.
But then I cannot see the data from cqlsh it gives me no result. After i have imported the data from incorrect-id-folder i see the data
nodetool import keyspace_name table_name /full_path_of_old(incorrect)_folder.
now my questions are like:
First question; before i have restarted the nodes how can i search the data although there is a mismatch between system_schema.tables and actual directories in all nodes.
Second one; is nodetool import a safe way to load data from incorrect folder on a write heavy system. Because I cannot be sure if I miss any data during the import operation. Or do i need to run a repair for those tables instead of import? In my opinion that may not work because i cannot see any data before nodetool import.
Thanks again.
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 1:17:11 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Cc: Bowen Song
Subject: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
It sounds like you have done some concurrent table creation/deletion in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients), which resulted in this mismatch. After you restarted the node, Cassandra corrected it by discarding the old table ID and any data associated with it. This is the expected behaviour. This issue has already been fixed, and you can safely delete the data directory with the incorrect table ID as it is no longer used by Cassandra. You should now run a full repair on this node to ensure it has all the data it owns. If you are absolutely certain that the table with different IDs have identical schema, and the gc_grace_seconds hasn't past, you may move the data from the wrong data directory to the correct data directory, and then restart the node or run "nodetool refresh <keyspace> <table>" on the node before running the full repair, this may save you some streaming time. However, if the table schema is different, this may cause a havoc.
On 18/01/2024 05:21, ENES ATABERK wrote:
Hi all,
we have detected that table-uuid in linux file directory is different from system_schema.tables id.
I have executed nodetool describe cluster and see only one schema version in the cluster.
How we can fix this issue do anyone has any idea? Restarting the nodes only create a new empty directory with name system_schema.tables id directory but in this case i have two directories old one has sstables with incorrect uuid new one has correct uuid but empty.
thanks in advance
[https://s.turkcell.com.tr/SiteAssets/Genel/mail/mail-imza.jpg] <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
________________________________
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted.
TURKCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of TURKCELL.
This e-mail has been scanned for all known computer viruses.
Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by Bowen Song via user <us...@cassandra.apache.org>.
I know dropping a table and then creating a new table with the same name
can lead to that result, which is expected. If that wasn't what
happened, it may be a bug in Cassandra. If you can reproduce the
behaviour, you should raise a Jira ticket for it.
On 18/01/2024 14:44, ENES ATABERK wrote:
>
> It has same mismatch id in all nodes not just one node.
>
>
> ------------------------------------------------------------------------
> *From:* Bowen Song via user <us...@cassandra.apache.org>
> *Sent:* Thursday, January 18, 2024 3:18:11 PM
> *To:* user@cassandra.apache.org
> *Cc:* Bowen Song
> *Subject:* COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and
> table uuid on disk mismatch
>
> Was the table ID mismatching only on one node or all nodes?
> Mismatching on one node is usually the result of a racing condition,
> but on all nodes isn't. The solution I mentioned earlier only applies
> to the one node situation.
>
>
> On 18/01/2024 13:14, ENES ATABERK wrote:
>>
>> Hi all,
>>
>> Thanks for your responses.
>>
>> The version is Cassandra 4.1.3
>>
>> After I restarted all the nodes one-by-one cassandra created
>> corrected-id folder and keep the incorrect one as you said.
>>
>> But then I cannot see the data from cqlsh it gives me no result.
>> After i have imported the data from incorrect-id-folder i see the data
>>
>>
>> nodetool import keyspace_name table_name
>> /full_path_of_old(incorrect)_folder.
>>
>>
>> now my questions are like:
>>
>> First question; before i have restarted the nodes how can i search
>> the data although there is a mismatch between system_schema.tables
>> and actual directories in all nodes.
>>
>>
>> Second one; is nodetool import a safe way to load data from incorrect
>> folder on a write heavy system. Because I cannot be sure if I miss
>> any data during the import operation. Or do i need to run a repair
>> for those tables instead of import? In my opinion that may not work
>> because i cannot see any data before nodetool import.
>>
>>
>> Thanks again.
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Bowen Song via user <us...@cassandra.apache.org>
>> *Sent:* Thursday, January 18, 2024 1:17:11 PM
>> *To:* user@cassandra.apache.org
>> *Cc:* Bowen Song
>> *Subject:* COMMERCIAL:Re: system_schema.tables id and table uuid on
>> disk mismatch
>>
>> It sounds like you have done some concurrent table creation/deletion
>> in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients),
>> which resulted in this mismatch. After you restarted the node,
>> Cassandra corrected it by discarding the old table ID and any data
>> associated with it. This is the expected behaviour. This issue has
>> already been fixed, and you can safely delete the data directory with
>> the incorrect table ID as it is no longer used by Cassandra. You
>> should now run a full repair on this node to ensure it has all the
>> data it owns. If you are /absolutely/ certain that the table with
>> different IDs have identical schema, and the gc_grace_seconds hasn't
>> past, you may move the data from the wrong data directory to the
>> correct data directory, and then restart the node or run "nodetool
>> refresh <keyspace> <table>" on the node before running the full
>> repair, this may save you some streaming time. However, if the table
>> schema is different, this may cause a havoc.
>>
>>
>> On 18/01/2024 05:21, ENES ATABERK wrote:
>>>
>>> Hi all,
>>>
>>> we have detected that table-uuid in linux file directory is
>>> different from system_schema.tables id.
>>>
>>> I have executed nodetool describe cluster and see only one schema
>>> version in the cluster.
>>>
>>> How we can fix this issue do anyone has any idea? Restarting the
>>> nodes only create a new empty directory with
>>> name system_schema.tables id directory but in this case i have two
>>> directories old one has sstables with incorrect uuid new one has
>>> correct uuid but empty.
>>>
>>> thanks in advance
>>>
>>>
>>>
>>>
>>> <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
>>>
>>> Bu elektronik posta ve onunla iletilen butun dosyalar sadece
>>> gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel
>>> kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz
>>> bu elektronik postanin icerigini aciklamaniz, kopyalamaniz,
>>> yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik
>>> postayi derhal silmeniz gerekmektedir.
>>>
>>> TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz
>>> oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu
>>> bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden,
>>> alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki
>>> gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini
>>> yansitmayabilir
>>>
>>> Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
>>>
>>> ------------------------------------------------------------------------
>>>
>>> This e-mail and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom they
>>> are addressed. If you are not the intended recipient you are hereby
>>> notified that any dissemination, forwarding, copying or use of any
>>> of the information is strictly prohibited, and the e-mail should
>>> immediately be deleted.
>>>
>>> TURKCELL makes no warranty as to the accuracy or completeness of any
>>> information contained in this message and hereby excludes any
>>> liability of any kind for the information contained therein or for
>>> the information transmission, reception, storage or use of such in
>>> any way whatsoever. The opinions expressed in this message belong to
>>> sender alone and may not necessarily reflect the opinions of TURKCELL.
>>>
>>> This e-mail has been scanned for all known computer viruses.
>>>
>>
>
Re: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by ENES ATABERK <en...@turkcell.com.tr>.
It has same mismatch id in all nodes not just one node.
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 3:18:11 PM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: COMMERCIAL:Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Was the table ID mismatching only on one node or all nodes? Mismatching on one node is usually the result of a racing condition, but on all nodes isn't. The solution I mentioned earlier only applies to the one node situation.
On 18/01/2024 13:14, ENES ATABERK wrote:
Hi all,
Thanks for your responses.
The version is Cassandra 4.1.3
After I restarted all the nodes one-by-one cassandra created corrected-id folder and keep the incorrect one as you said.
But then I cannot see the data from cqlsh it gives me no result. After i have imported the data from incorrect-id-folder i see the data
nodetool import keyspace_name table_name /full_path_of_old(incorrect)_folder.
now my questions are like:
First question; before i have restarted the nodes how can i search the data although there is a mismatch between system_schema.tables and actual directories in all nodes.
Second one; is nodetool import a safe way to load data from incorrect folder on a write heavy system. Because I cannot be sure if I miss any data during the import operation. Or do i need to run a repair for those tables instead of import? In my opinion that may not work because i cannot see any data before nodetool import.
Thanks again.
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 1:17:11 PM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Cc: Bowen Song
Subject: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
It sounds like you have done some concurrent table creation/deletion in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients), which resulted in this mismatch. After you restarted the node, Cassandra corrected it by discarding the old table ID and any data associated with it. This is the expected behaviour. This issue has already been fixed, and you can safely delete the data directory with the incorrect table ID as it is no longer used by Cassandra. You should now run a full repair on this node to ensure it has all the data it owns. If you are absolutely certain that the table with different IDs have identical schema, and the gc_grace_seconds hasn't past, you may move the data from the wrong data directory to the correct data directory, and then restart the node or run "nodetool refresh <keyspace> <table>" on the node before running the full repair, this may save you some streaming time. However, if the table schema is different, this may cause a havoc.
On 18/01/2024 05:21, ENES ATABERK wrote:
Hi all,
we have detected that table-uuid in linux file directory is different from system_schema.tables id.
I have executed nodetool describe cluster and see only one schema version in the cluster.
How we can fix this issue do anyone has any idea? Restarting the nodes only create a new empty directory with name system_schema.tables id directory but in this case i have two directories old one has sstables with incorrect uuid new one has correct uuid but empty.
thanks in advance
[https://s.turkcell.com.tr/SiteAssets/Genel/mail/mail-imza.jpg] <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
________________________________
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted.
TURKCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of TURKCELL.
This e-mail has been scanned for all known computer viruses.
Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by Bowen Song via user <us...@cassandra.apache.org>.
Was the table ID mismatching only on one node or all nodes? Mismatching
on one node is usually the result of a racing condition, but on all
nodes isn't. The solution I mentioned earlier only applies to the one
node situation.
On 18/01/2024 13:14, ENES ATABERK wrote:
>
> Hi all,
>
> Thanks for your responses.
>
> The version is Cassandra 4.1.3
>
> After I restarted all the nodes one-by-one cassandra created
> corrected-id folder and keep the incorrect one as you said.
>
> But then I cannot see the data from cqlsh it gives me no result. After
> i have imported the data from incorrect-id-folder i see the data
>
>
> nodetool import keyspace_name table_name
> /full_path_of_old(incorrect)_folder.
>
>
> now my questions are like:
>
> First question; before i have restarted the nodes how can i search the
> data although there is a mismatch between system_schema.tables and
> actual directories in all nodes.
>
>
> Second one; is nodetool import a safe way to load data from incorrect
> folder on a write heavy system. Because I cannot be sure if I miss any
> data during the import operation. Or do i need to run a repair for
> those tables instead of import? In my opinion that may not work
> because i cannot see any data before nodetool import.
>
>
> Thanks again.
>
>
> ------------------------------------------------------------------------
> *From:* Bowen Song via user <us...@cassandra.apache.org>
> *Sent:* Thursday, January 18, 2024 1:17:11 PM
> *To:* user@cassandra.apache.org
> *Cc:* Bowen Song
> *Subject:* COMMERCIAL:Re: system_schema.tables id and table uuid on
> disk mismatch
>
> It sounds like you have done some concurrent table creation/deletion
> in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients),
> which resulted in this mismatch. After you restarted the node,
> Cassandra corrected it by discarding the old table ID and any data
> associated with it. This is the expected behaviour. This issue has
> already been fixed, and you can safely delete the data directory with
> the incorrect table ID as it is no longer used by Cassandra. You
> should now run a full repair on this node to ensure it has all the
> data it owns. If you are /absolutely/ certain that the table with
> different IDs have identical schema, and the gc_grace_seconds hasn't
> past, you may move the data from the wrong data directory to the
> correct data directory, and then restart the node or run "nodetool
> refresh <keyspace> <table>" on the node before running the full
> repair, this may save you some streaming time. However, if the table
> schema is different, this may cause a havoc.
>
>
> On 18/01/2024 05:21, ENES ATABERK wrote:
>>
>> Hi all,
>>
>> we have detected that table-uuid in linux file directory is different
>> from system_schema.tables id.
>>
>> I have executed nodetool describe cluster and see only one schema
>> version in the cluster.
>>
>> How we can fix this issue do anyone has any idea? Restarting the
>> nodes only create a new empty directory with
>> name system_schema.tables id directory but in this case i have two
>> directories old one has sstables with incorrect uuid new one has
>> correct uuid but empty.
>>
>> thanks in advance
>>
>>
>>
>>
>> <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
>>
>> Bu elektronik posta ve onunla iletilen butun dosyalar sadece
>> gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel
>> kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz
>> bu elektronik postanin icerigini aciklamaniz, kopyalamaniz,
>> yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik
>> postayi derhal silmeniz gerekmektedir.
>>
>> TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz
>> oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu
>> bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden,
>> alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki
>> gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini
>> yansitmayabilir
>>
>> Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
>>
>> ------------------------------------------------------------------------
>>
>> This e-mail and any files transmitted with it are confidential and
>> intended solely for the use of the individual or entity to whom they
>> are addressed. If you are not the intended recipient you are hereby
>> notified that any dissemination, forwarding, copying or use of any of
>> the information is strictly prohibited, and the e-mail should
>> immediately be deleted.
>>
>> TURKCELL makes no warranty as to the accuracy or completeness of any
>> information contained in this message and hereby excludes any
>> liability of any kind for the information contained therein or for
>> the information transmission, reception, storage or use of such in
>> any way whatsoever. The opinions expressed in this message belong to
>> sender alone and may not necessarily reflect the opinions of TURKCELL.
>>
>> This e-mail has been scanned for all known computer viruses.
>>
>
Re: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
Posted by ENES ATABERK <en...@turkcell.com.tr>.
Hi all,
Thanks for your responses.
The version is Cassandra 4.1.3
After I restarted all the nodes one-by-one cassandra created corrected-id folder and keep the incorrect one as you said.
But then I cannot see the data from cqlsh it gives me no result. After i have imported the data from incorrect-id-folder i see the data
nodetool import keyspace_name table_name /full_path_of_old(incorrect)_folder.
now my questions are like:
First question; before i have restarted the nodes how can i search the data although there is a mismatch between system_schema.tables and actual directories in all nodes.
Second one; is nodetool import a safe way to load data from incorrect folder on a write heavy system. Because I cannot be sure if I miss any data during the import operation. Or do i need to run a repair for those tables instead of import? In my opinion that may not work because i cannot see any data before nodetool import.
Thanks again.
________________________________
From: Bowen Song via user <us...@cassandra.apache.org>
Sent: Thursday, January 18, 2024 1:17:11 PM
To: user@cassandra.apache.org
Cc: Bowen Song
Subject: COMMERCIAL:Re: system_schema.tables id and table uuid on disk mismatch
It sounds like you have done some concurrent table creation/deletion in the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients), which resulted in this mismatch. After you restarted the node, Cassandra corrected it by discarding the old table ID and any data associated with it. This is the expected behaviour. This issue has already been fixed, and you can safely delete the data directory with the incorrect table ID as it is no longer used by Cassandra. You should now run a full repair on this node to ensure it has all the data it owns. If you are absolutely certain that the table with different IDs have identical schema, and the gc_grace_seconds hasn't past, you may move the data from the wrong data directory to the correct data directory, and then restart the node or run "nodetool refresh <keyspace> <table>" on the node before running the full repair, this may save you some streaming time. However, if the table schema is different, this may cause a havoc.
On 18/01/2024 05:21, ENES ATABERK wrote:
Hi all,
we have detected that table-uuid in linux file directory is different from system_schema.tables id.
I have executed nodetool describe cluster and see only one schema version in the cluster.
How we can fix this issue do anyone has any idea? Restarting the nodes only create a new empty directory with name system_schema.tables id directory but in this case i have two directories old one has sstables with incorrect uuid new one has correct uuid but empty.
thanks in advance
[https://s.turkcell.com.tr/SiteAssets/Genel/mail/mail-imza.jpg] <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
Bu elektronik posta ve onunla iletilen butun dosyalar sadece gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz bu elektronik postanin icerigini aciklamaniz, kopyalamaniz, yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik postayi derhal silmeniz gerekmektedir.
TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca gonderen kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
________________________________
This e-mail and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you are not the intended recipient you are hereby notified that any dissemination, forwarding, copying or use of any of the information is strictly prohibited, and the e-mail should immediately be deleted.
TURKCELL makes no warranty as to the accuracy or completeness of any information contained in this message and hereby excludes any liability of any kind for the information contained therein or for the information transmission, reception, storage or use of such in any way whatsoever. The opinions expressed in this message belong to sender alone and may not necessarily reflect the opinions of TURKCELL.
This e-mail has been scanned for all known computer viruses.
Re: system_schema.tables id and table uuid on disk mismatch
Posted by Bowen Song via user <us...@cassandra.apache.org>.
It sounds like you have done some concurrent table creation/deletion in
the past (e.g. CREATE TABLE IF NOT EXISTS from multiple clients), which
resulted in this mismatch. After you restarted the node, Cassandra
corrected it by discarding the old table ID and any data associated with
it. This is the expected behaviour. This issue has already been fixed,
and you can safely delete the data directory with the incorrect table ID
as it is no longer used by Cassandra. You should now run a full repair
on this node to ensure it has all the data it owns. If you are
/absolutely/ certain that the table with different IDs have identical
schema, and the gc_grace_seconds hasn't past, you may move the data from
the wrong data directory to the correct data directory, and then restart
the node or run "nodetool refresh <keyspace> <table>" on the node before
running the full repair, this may save you some streaming time. However,
if the table schema is different, this may cause a havoc.
On 18/01/2024 05:21, ENES ATABERK wrote:
>
> Hi all,
>
> we have detected that table-uuid in linux file directory is different
> from system_schema.tables id.
>
> I have executed nodetool describe cluster and see only one schema
> version in the cluster.
>
> How we can fix this issue do anyone has any idea? Restarting the nodes
> only create a new empty directory with name system_schema.tables id
> directory but in this case i have two directories old one has sstables
> with incorrect uuid new one has correct uuid but empty.
>
> thanks in advance
>
>
>
>
> <https://www.turkcell.com.tr/kurumsal/isturkcell-plus>
>
> Bu elektronik posta ve onunla iletilen butun dosyalar sadece
> gondericisi tarafindan almasi amaclanan yetkili gercek ya da tuzel
> kisinin kullanimi icindir. Eger soz konusu yetkili alici degilseniz bu
> elektronik postanin icerigini aciklamaniz, kopyalamaniz,
> yonlendirmeniz ve kullanmaniz kesinlikle yasaktir ve bu elektronik
> postayi derhal silmeniz gerekmektedir.
>
> TURKCELL bu mesajin icerdigi bilgilerin doğruluğu veya eksiksiz oldugu
> konusunda herhangi bir garanti vermemektedir. Bu nedenle bu bilgilerin
> ne sekilde olursa olsun iceriginden, iletilmesinden, alinmasindan ve
> saklanmasindan sorumlu degildir. Bu mesajdaki gorusler yalnizca
> gonderen kisiye aittir ve TURKCELLin goruslerini yansitmayabilir
>
> Bu e-posta bilinen butun bilgisayar viruslerine karsi taranmistir.
>
> ------------------------------------------------------------------------
>
> This e-mail and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom they
> are addressed. If you are not the intended recipient you are hereby
> notified that any dissemination, forwarding, copying or use of any of
> the information is strictly prohibited, and the e-mail should
> immediately be deleted.
>
> TURKCELL makes no warranty as to the accuracy or completeness of any
> information contained in this message and hereby excludes any
> liability of any kind for the information contained therein or for the
> information transmission, reception, storage or use of such in any way
> whatsoever. The opinions expressed in this message belong to sender
> alone and may not necessarily reflect the opinions of TURKCELL.
>
> This e-mail has been scanned for all known computer viruses.
>