You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Berenguer Blasi (Jira)" <ji...@apache.org> on 2021/05/24 07:45:00 UTC

[jira] [Comment Edited] (CASSANDRA-12519) dtest failure in offline_tools_test.TestOfflineTools.sstableofflinerelevel_test

    [ https://issues.apache.org/jira/browse/CASSANDRA-12519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17350284#comment-17350284 ] 

Berenguer Blasi edited comment on CASSANDRA-12519 at 5/24/21, 7:44 AM:
-----------------------------------------------------------------------

Happened again [here|https://ci-cassandra.apache.org/job/Cassandra-4.0/53/testReport/junit/dtest.offline_tools_test/TestOfflineTools/test_sstableofflinerelevel/] and [here|https://ci-cassandra.apache.org/job/Cassandra-4.0/54/testReport/junit/dtest-offheap.offline_tools_test/TestOfflineTools/test_sstableofflinerelevel/]

{noformat}
Regression

dtest.offline_tools_test.TestOfflineTools.test_sstableofflinerelevel (from Cassandra dtests)
Failing for the past 1 build (Since Unstable#53 )
Took 3 min 28 sec.
Failed 1 times in the last 4 runs. Flakiness: 33%, Stability: 75%
Error Message

ccmlib.node.ToolError: Subprocess sstableoflinerelevel on keyspace1 : standard1 exited with non-zero status; exit status: 1;  stderr: WARN  11:30:58,846 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB Exception in thread "main" java.lang.RuntimeException: Failed to list files in /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-y84fag44/test/node1/data0/keyspace1/standard1-b696b6e0baf011eb91ff8b04c35d7610  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)  at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:604)  at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:892)  at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:857)  at org.apache.cassandra.tools.SSTableOfflineRelevel.main(SSTableOfflineRelevel.java:104) Caused by: org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException: Some records failed verification. See earlier in log for details.  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:136)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:127)  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:95)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)  ... 4 more

Stacktrace

self = <offline_tools_test.TestOfflineTools object at 0x7fd9db672a90>

    def test_sstableofflinerelevel(self):
        """
            Generate sstables of varying levels.
            Reset sstables to L0 with sstablelevelreset
            Run sstableofflinerelevel and ensure tables are promoted correctly
            Also test a variety of bad inputs including nonexistent keyspace and sstables
            @since 2.1.5
            @jira_ticket CASSANRDA-8031
            """
        cluster = self.cluster
        cluster.set_configuration_options(values={'compaction_throughput_mb_per_sec': 0})
        cluster.populate(1).start()
        node1 = cluster.nodelist()[0]
    
        # NOTE - As of now this does not return when it encounters Exception and causes test to hang, temporarily commented out
        # test by trying to run on nonexistent keyspace
        # cluster.stop(gently=False)
        # output, error, rc = node1.run_sstableofflinerelevel("keyspace1", "standard1", output=True)
        # assert "java.lang.IllegalArgumentException: Unknown keyspace/columnFamily keyspace1.standard1" in error
        # # this should return exit code 1
        # assert rc, 1 == msg=str(rc)
        # cluster.start()
    
        # now test by generating keyspace but not flushing sstables
    
        node1.stress(['write', 'n=1', 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1024)',
                      '-rate', 'threads=8'])
    
        cluster.stop(gently=False)
        try:
            output, error, _ = node1.run_sstableofflinerelevel("keyspace1", "standard1")
        except ToolError as e:
            assert re.search("No sstables to relevel for keyspace1.standard1", e.stdout)
            assert e.exit_status == 1, str(e.exit_status)
    
        # test by flushing (sstable should be level 0)
        cluster.start()
        session = self.patient_cql_connection(node1)
        logger.debug("Altering compaction strategy to LCS")
        session.execute("ALTER TABLE keyspace1.standard1 with compaction={'class': 'LeveledCompactionStrategy', 'sstable_size_in_mb':1, 'enabled':'false'};")
    
        node1.stress(['write', 'n=1K', 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1024)',
                      '-rate', 'threads=8'])
    
        node1.flush()
        cluster.stop()
    
        output, _, rc = node1.run_sstableofflinerelevel("keyspace1", "standard1")
        assert re.search("L0=1", output)
        assert rc == 0, str(rc)
    
        cluster.start()
        node1.nodetool('enableautocompaction keyspace1 standard1')
        # test by loading large amount data so we have multiple sstables
        # must write enough to create more than just L1 sstables
        keys = 8 * cluster.data_dir_count
        node1.stress(['write', 'n={0}K'.format(keys), 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1200)',
                      '-rate', 'threads=8'])
    
        node1.flush()
        logger.debug("Waiting for compactions to finish")
        self.wait_for_compactions(node1)
        logger.debug("Stopping node")
        cluster.stop()
        logger.debug("Done stopping node")
    
        # Let's reset all sstables to L0
        logger.debug("Getting initial levels")
        initial_levels = list(self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"])))
        assert [] != initial_levels
        logger.debug('initial_levels:')
        logger.debug(initial_levels)
        logger.debug("Running sstablelevelreset")
        node1.run_sstablelevelreset("keyspace1", "standard1")
        logger.debug("Getting final levels")
        final_levels = list(self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"])))
        assert [] != final_levels
        logger.debug('final levels:')
        logger.debug(final_levels)
    
        # let's make sure there was at least 3 levels (L0, L1 and L2)
        assert max(initial_levels) > 1
        # let's check all sstables are on L0 after sstablelevelreset
        assert max(final_levels) == 0
    
        # time to relevel sstables
        logger.debug("Getting initial levels")
        initial_levels = self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"]))
        logger.debug("Running sstableofflinerelevel")
>       output, error, _ = node1.run_sstableofflinerelevel("keyspace1", "standard1")

offline_tools_test.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/src/ccm/ccmlib/node.py:1368: in run_sstableofflinerelevel
    return handle_external_tool_process(p, "sstableoflinerelevel on {} : {}".format(keyspace, cf))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

process = <subprocess.Popen object at 0x7fd9d9002100>
cmd_args = 'sstableoflinerelevel on keyspace1 : standard1'

    def handle_external_tool_process(process, cmd_args):
        out, err = process.communicate()
        if (out is not None) and isinstance(out, bytes):
            out = out.decode()
        if (err is not None) and isinstance(err, bytes):
            err = err.decode()
        rc = process.returncode
    
        if rc != 0:
>           raise ToolError(cmd_args, rc, out, err)
E           ccmlib.node.ToolError: Subprocess sstableoflinerelevel on keyspace1 : standard1 exited with non-zero status; exit status: 1; 
E           stderr: WARN  11:30:58,846 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           Exception in thread "main" java.lang.RuntimeException: Failed to list files in /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-y84fag44/test/node1/data0/keyspace1/standard1-b696b6e0baf011eb91ff8b04c35d7610
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
E           	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:604)
E           	at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:892)
E           	at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:857)
E           	at org.apache.cassandra.tools.SSTableOfflineRelevel.main(SSTableOfflineRelevel.java:104)
E           Caused by: org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException: Some records failed verification. See earlier in log for details.
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:136)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:127)
E           	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
E           	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
E           	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
E           	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
E           	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
E           	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
E           	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
E           	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
E           	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:95)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
E           	... 4 more

../venv/src/ccm/ccmlib/node.py:2303: ToolError
{noformat}



was (Author: bereng):
Happened again [here|https://ci-cassandra.apache.org/job/Cassandra-4.0/53/testReport/junit/dtest.offline_tools_test/TestOfflineTools/test_sstableofflinerelevel/]

{noformat}
Regression

dtest.offline_tools_test.TestOfflineTools.test_sstableofflinerelevel (from Cassandra dtests)
Failing for the past 1 build (Since Unstable#53 )
Took 3 min 28 sec.
Failed 1 times in the last 4 runs. Flakiness: 33%, Stability: 75%
Error Message

ccmlib.node.ToolError: Subprocess sstableoflinerelevel on keyspace1 : standard1 exited with non-zero status; exit status: 1;  stderr: WARN  11:30:58,846 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB Exception in thread "main" java.lang.RuntimeException: Failed to list files in /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-y84fag44/test/node1/data0/keyspace1/standard1-b696b6e0baf011eb91ff8b04c35d7610  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)  at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:604)  at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:892)  at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:857)  at org.apache.cassandra.tools.SSTableOfflineRelevel.main(SSTableOfflineRelevel.java:104) Caused by: org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException: Some records failed verification. See earlier in log for details.  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:136)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:127)  at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)  at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)  at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)  at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)  at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)  at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)  at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)  at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)  at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:95)  at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)  ... 4 more

Stacktrace

self = <offline_tools_test.TestOfflineTools object at 0x7fd9db672a90>

    def test_sstableofflinerelevel(self):
        """
            Generate sstables of varying levels.
            Reset sstables to L0 with sstablelevelreset
            Run sstableofflinerelevel and ensure tables are promoted correctly
            Also test a variety of bad inputs including nonexistent keyspace and sstables
            @since 2.1.5
            @jira_ticket CASSANRDA-8031
            """
        cluster = self.cluster
        cluster.set_configuration_options(values={'compaction_throughput_mb_per_sec': 0})
        cluster.populate(1).start()
        node1 = cluster.nodelist()[0]
    
        # NOTE - As of now this does not return when it encounters Exception and causes test to hang, temporarily commented out
        # test by trying to run on nonexistent keyspace
        # cluster.stop(gently=False)
        # output, error, rc = node1.run_sstableofflinerelevel("keyspace1", "standard1", output=True)
        # assert "java.lang.IllegalArgumentException: Unknown keyspace/columnFamily keyspace1.standard1" in error
        # # this should return exit code 1
        # assert rc, 1 == msg=str(rc)
        # cluster.start()
    
        # now test by generating keyspace but not flushing sstables
    
        node1.stress(['write', 'n=1', 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1024)',
                      '-rate', 'threads=8'])
    
        cluster.stop(gently=False)
        try:
            output, error, _ = node1.run_sstableofflinerelevel("keyspace1", "standard1")
        except ToolError as e:
            assert re.search("No sstables to relevel for keyspace1.standard1", e.stdout)
            assert e.exit_status == 1, str(e.exit_status)
    
        # test by flushing (sstable should be level 0)
        cluster.start()
        session = self.patient_cql_connection(node1)
        logger.debug("Altering compaction strategy to LCS")
        session.execute("ALTER TABLE keyspace1.standard1 with compaction={'class': 'LeveledCompactionStrategy', 'sstable_size_in_mb':1, 'enabled':'false'};")
    
        node1.stress(['write', 'n=1K', 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1024)',
                      '-rate', 'threads=8'])
    
        node1.flush()
        cluster.stop()
    
        output, _, rc = node1.run_sstableofflinerelevel("keyspace1", "standard1")
        assert re.search("L0=1", output)
        assert rc == 0, str(rc)
    
        cluster.start()
        node1.nodetool('enableautocompaction keyspace1 standard1')
        # test by loading large amount data so we have multiple sstables
        # must write enough to create more than just L1 sstables
        keys = 8 * cluster.data_dir_count
        node1.stress(['write', 'n={0}K'.format(keys), 'no-warmup',
                      '-schema', 'replication(factor=1)',
                      '-col', 'n=FIXED(10)', 'SIZE=FIXED(1200)',
                      '-rate', 'threads=8'])
    
        node1.flush()
        logger.debug("Waiting for compactions to finish")
        self.wait_for_compactions(node1)
        logger.debug("Stopping node")
        cluster.stop()
        logger.debug("Done stopping node")
    
        # Let's reset all sstables to L0
        logger.debug("Getting initial levels")
        initial_levels = list(self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"])))
        assert [] != initial_levels
        logger.debug('initial_levels:')
        logger.debug(initial_levels)
        logger.debug("Running sstablelevelreset")
        node1.run_sstablelevelreset("keyspace1", "standard1")
        logger.debug("Getting final levels")
        final_levels = list(self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"])))
        assert [] != final_levels
        logger.debug('final levels:')
        logger.debug(final_levels)
    
        # let's make sure there was at least 3 levels (L0, L1 and L2)
        assert max(initial_levels) > 1
        # let's check all sstables are on L0 after sstablelevelreset
        assert max(final_levels) == 0
    
        # time to relevel sstables
        logger.debug("Getting initial levels")
        initial_levels = self.get_levels(node1.run_sstablemetadata(keyspace="keyspace1", column_families=["standard1"]))
        logger.debug("Running sstableofflinerelevel")
>       output, error, _ = node1.run_sstableofflinerelevel("keyspace1", "standard1")

offline_tools_test.py:204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../venv/src/ccm/ccmlib/node.py:1368: in run_sstableofflinerelevel
    return handle_external_tool_process(p, "sstableoflinerelevel on {} : {}".format(keyspace, cf))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

process = <subprocess.Popen object at 0x7fd9d9002100>
cmd_args = 'sstableoflinerelevel on keyspace1 : standard1'

    def handle_external_tool_process(process, cmd_args):
        out, err = process.communicate()
        if (out is not None) and isinstance(out, bytes):
            out = out.decode()
        if (err is not None) and isinstance(err, bytes):
            err = err.decode()
        rc = process.returncode
    
        if rc != 0:
>           raise ToolError(cmd_args, rc, out, err)
E           ccmlib.node.ToolError: Subprocess sstableoflinerelevel on keyspace1 : standard1 exited with non-zero status; exit status: 1; 
E           stderr: WARN  11:30:58,846 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           WARN  11:30:58,852 Max sstable size of 1MB is configured for standard1.standard1.  Testing done for CASSANDRA-5727 indicates that performance improves up to 160MB
E           Exception in thread "main" java.lang.RuntimeException: Failed to list files in /home/cassandra/cassandra/cassandra-dtest/tmp/dtest-y84fag44/test/node1/data0/keyspace1/standard1-b696b6e0baf011eb91ff8b04c35d7610
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:77)
E           	at org.apache.cassandra.db.lifecycle.LifecycleTransaction.getFiles(LifecycleTransaction.java:604)
E           	at org.apache.cassandra.db.Directories$SSTableLister.filter(Directories.java:892)
E           	at org.apache.cassandra.db.Directories$SSTableLister.list(Directories.java:857)
E           	at org.apache.cassandra.tools.SSTableOfflineRelevel.main(SSTableOfflineRelevel.java:104)
E           Caused by: org.apache.cassandra.db.lifecycle.LogTransaction$CorruptTransactionLogException: Some records failed verification. See earlier in log for details.
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.readTxnLog(LogAwareFileLister.java:136)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.classifyFiles(LogAwareFileLister.java:127)
E           	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
E           	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
E           	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384)
E           	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
E           	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
E           	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
E           	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
E           	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
E           	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:485)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.innerList(LogAwareFileLister.java:95)
E           	at org.apache.cassandra.db.lifecycle.LogAwareFileLister.list(LogAwareFileLister.java:73)
E           	... 4 more

../venv/src/ccm/ccmlib/node.py:2303: ToolError
{noformat}


> dtest failure in offline_tools_test.TestOfflineTools.sstableofflinerelevel_test
> -------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-12519
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12519
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Test/dtest/python
>            Reporter: Sean McCarthy
>            Priority: Normal
>             Fix For: 4.0-rc2, 4.0, 3.0.x, 3.11.x, 4.x
>
>         Attachments: node1.log, node1_debug.log, node1_gc.log
>
>
> example failure: http://cassci.datastax.com/job/trunk_offheap_dtest/379/testReport/offline_tools_test/TestOfflineTools/sstableofflinerelevel_test/
> {code}
> Stacktrace
>   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
>     testMethod()
>   File "/home/automaton/cassandra-dtest/offline_tools_test.py", line 209, in sstableofflinerelevel_test
>     self.assertGreater(max(final_levels), 1)
>   File "/usr/lib/python2.7/unittest/case.py", line 942, in assertGreater
>     self.fail(self._formatMessage(msg, standardMsg))
>   File "/usr/lib/python2.7/unittest/case.py", line 410, in fail
>     raise self.failureException(msg)
> "1 not greater than 1
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org