You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jon Haddad (Jira)" <ji...@apache.org> on 2026/03/04 02:16:00 UTC
[jira] [Created] (CASSANDRA-21197) import not importing resulting in data loss with analytics jobs
Jon Haddad created CASSANDRA-21197:
--------------------------------------
Summary: import not importing resulting in data loss with analytics jobs
Key: CASSANDRA-21197
URL: https://issues.apache.org/jira/browse/CASSANDRA-21197
Project: Apache Cassandra
Issue Type: Bug
Components: Analytics Library
Reporter: Jon Haddad
When evaluating the analytics bulk writer I found jobs were reported as successful, but the data wasn't being correctly imported. I'm testing using C* 5.0.6, sidecar trunk (as of yesterday), and the latest analytics code as of... recent.
I've traced the network and filesystem calls and have found this is the series of events:
1. Spark job runs
2. data lands on disk from sidecar
3. import is called, C* says nothing to import
4. sidecar then deletes the data files
resulting in all my data getting deleted off disk, without import happening. I have tested this dozens of times a day for almost a week and it's happened 100% of the time.
I haven't yet determined why Cassandra doesn't import anything, but given the nature of the issue I'm hoping more eyes on this will help. It's possible there's something specific about my setup that's causing this issue - I know there are quite a few tests around sidecar, so I'm surprised it's happening.
That said, if C* isn't correctly importing data, it should have a way of telling sidecar that so sidecar doesn't delete the results of a bulk write job.
{*}Note{*}: the names of the files might not match up here, I've done this over several days now with about a dozen clusters and 100 spark jobs.
[Spark job runs|[https://github.com/rustyrazorblade/easy-db-lab/blob/main/bin/submit-direct-bulk-writer]]. The data files are written to disk, then renamed. I've captured that several ways, the easiest way to see it is here for the rename, captured with sysdig:
{noformat}
sudo sysdig "evt.category=file and (proc.pid=24272 or proc.pid=30444)" | grep 'cassandra/import'{noformat}
Here's the relevant output, where the vertx process (sidecar) performs the rename to the expected data file name:
{noformat}
2198732 01:17:36.437748828 1 vert.x-internal (30642) < rename res=0 oldpath=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Index.db16346060661306473655.tmp newpath=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Index.db
2199993 01:17:36.450173069 6 vert.x-internal (30635) < rename res=0 oldpath=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Filter.db4989982398684709072.tmp newpath=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Filter.db{noformat}
Process 30528 (cassandra) import is called. I captured the filesystem event where it receives 10 entries:
{noformat}
sudo strace -p 30528 -e trace=getdents64 -y 2>&1 | grep import
getdents64(402</mnt/db1/cassandra/import/0-0-28c91aa3-fcae-4c97-bf5a-e520f070e1f9-a0a1bdd0-176b-11f1-bc8d-55a3317257c0/bulk_test/data>, 0x7176a803a0c0 /* 10 entries */, 32768) = 392{noformat}
but the log entry says nothing is imported:
{noformat}
INFO [RMI TCP Connection(92)-127.0.0.1] 2026-03-04 01:44:12,773 SSTableImporter.java:80 - [af506331-6517-4461-a10f-3846baaf30c6] Loading new SSTables for bulk_test/data: Options{srcPaths='[/mnt/db1/cassandra/import/0-0-28c91aa3-fcae-4c97-bf5a-e520f070e1f9-a0a1bdd0-176b-11f1-bc8d-55a3317257c0/bulk_test/data]', resetLevel=true, clearRepaired=true, verifySSTables=true, verifyTokens=true, invalidateCaches=true, extendedVerify=false, copyData= false, failOnMissingIndex= false, validateIndexChecksum= true} INFO [RMI TCP Connection(92)-127.0.0.1] 2026-03-04 01:44:12,781 SSTableImporter.java:214 - [af506331-6517-4461-a10f-3846baaf30c6] No new SSTables were found for bulk_test/data{noformat}
sidecar then comes around and unlinks the files, resulting in data loss:
{noformat}
2248856 01:17:37.778334683 1 vert.x-internal (30642) < unlink res=0 path=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-CompressionInfo.db
2248866 01:17:37.778345865 1 vert.x-internal (30642) < newfstatat res=0 dirfd=-100(AT_FDCWD) path=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Statistics.db flags=256(AT_SYMLINK_NOFOLLOW)
2248868 01:17:37.778352848 1 vert.x-internal (30642) < newfstatat res=0 dirfd=-100(AT_FDCWD) path=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Statistics.db flags=256(AT_SYMLINK_NOFOLLOW)
2248875 01:17:37.778370298 1 vert.x-internal (30642) < unlink res=0 path=/mnt/db1/cassandra/import/0-0-1d50c5e6-8fbe-44c7-98ec-a06132e78c1f-e9293be0-1767-11f1-887e-0ff1d5cec701/bulk_test/data/oa-1-big-Statistics.db{noformat}
I haven't yet determined why Cassandra doesn't import the data. It sees the files in the listing, but there's no additional debug available to identify why it doesn't consider them valid.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org