You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Stefan Miklosovic (Jira)" <ji...@apache.org> on 2024/07/28 19:17:00 UTC
[jira] [Comment Edited] (CASSANDRA-19779) direct IO support is always evaluated to false upon the very first start of a node
[ https://issues.apache.org/jira/browse/CASSANDRA-19779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17869194#comment-17869194 ]
Stefan Miklosovic edited comment on CASSANDRA-19779 at 7/28/24 7:16 PM:
------------------------------------------------------------------------
Good news is that I fixed it for cassandra.yaml, the default config there is "legacy".
The bad news is that it fails these tests while using *cassandra_latest.yaml* profile
org.apache.cassandra.repair.ConcurrentIrWithPreviewFuzzTest
org.apache.cassandra.repair.FailedAckTest
org.apache.cassandra.repair.FailingRepairFuzzTest
org.apache.cassandra.repair.HappyPathFuzzTest
org.apache.cassandra.repair.SlowMessageFuzzTest
Exception it throws for all cases is
{code}
java.lang.UnsupportedOperationException
at java.base/java.nio.file.FileStore.getBlockSize(FileStore.java:158)
at org.apache.cassandra.io.util.FileUtils.getBlockSize(FileUtils.java:825)
at org.apache.cassandra.db.commitlog.DirectIOSegment$DirectIOSegmentBuilder.<init>(DirectIOSegment.java:166)
at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.createSegmentBuilder(AbstractCommitLogSegmentManager.java:135)
at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.start(AbstractCommitLogSegmentManager.java:154)
at org.apache.cassandra.db.commitlog.CommitLog.start(CommitLog.java:146)
at org.apache.cassandra.db.commitlog.CommitLog.restartUnsafe(CommitLog.java:563)
at org.apache.cassandra.ServerTestUtils.cleanupAndLeaveDirs(ServerTestUtils.java:148)
at org.apache.cassandra.ServerTestUtils.prepareServer(ServerTestUtils.java:106)
at org.apache.cassandra.cql3.CQLTester.prepareServer(CQLTester.java:360)
at org.apache.cassandra.cql3.CQLTester.setUpClass(CQLTester.java:411)
at org.apache.cassandra.cql3.CQLTester$InMemory.setUpClass(CQLTester.java:2876)
at org.apache.cassandra.repair.FuzzTestBase.setUpClass(FuzzTestBase.java:280)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
The reason this is happening is that it uses "InMemory.setUpClass" (5th line from the bottom). That does this:
{code}
@BeforeClass
public static void setUpClass()
{
fs = FileSystems.newGlobalInMemoryFileSystem();
CassandraRelevantProperties.IGNORE_MISSING_NATIVE_FILE_HINTS.setBoolean(true);
FileSystems.maybeCreateTmp();
CQLTester.setUpClass();
}
{code}
So, when it is "auto", previously, this failure was hidden, because commitlog dir was not created, hence "directIOSupported" was evaluated to false (the case described in the previous comment), so these tests have probably ever run with "legacy" only. So once we want to enable that, it does not play together, because here "FileUtils.getBlockSize" fails with UnsupportedOperationException because it is most probably not supported in "FileSystems.newGlobalInMemoryFileSystem();".
{code}
public DirectIOSegmentBuilder(AbstractCommitLogSegmentManager segmentManager)
{
this(segmentManager, FileUtils.getBlockSize(new File(segmentManager.storageDirectory)));
}
{code}
I am trying to figure out what to do. This also means that we have never fuzz tested "direct" mode for cassandra_latest.yaml because it the missing commitlog dir never triggered it.
was (Author: smiklosovic):
Good news is that I fixed it for cassandra.yaml, the default config there is "legacy".
The bad news is that it fails these tests while using *cassandra_latest.yaml* profile
org.apache.cassandra.repair.ConcurrentIrWithPreviewFuzzTest
org.apache.cassandra.repair.FailedAckTest
org.apache.cassandra.repair.FailingRepairFuzzTest
org.apache.cassandra.repair.HappyPathFuzzTest
org.apache.cassandra.repair.SlowMessageFuzzTest
Exception it throws for all cases is
{code}
java.lang.UnsupportedOperationException
at java.base/java.nio.file.FileStore.getBlockSize(FileStore.java:158)
at org.apache.cassandra.io.util.FileUtils.getBlockSize(FileUtils.java:825)
at org.apache.cassandra.db.commitlog.DirectIOSegment$DirectIOSegmentBuilder.<init>(DirectIOSegment.java:166)
at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.createSegmentBuilder(AbstractCommitLogSegmentManager.java:135)
at org.apache.cassandra.db.commitlog.AbstractCommitLogSegmentManager.start(AbstractCommitLogSegmentManager.java:154)
at org.apache.cassandra.db.commitlog.CommitLog.start(CommitLog.java:146)
at org.apache.cassandra.db.commitlog.CommitLog.restartUnsafe(CommitLog.java:563)
at org.apache.cassandra.ServerTestUtils.cleanupAndLeaveDirs(ServerTestUtils.java:148)
at org.apache.cassandra.ServerTestUtils.prepareServer(ServerTestUtils.java:106)
at org.apache.cassandra.cql3.CQLTester.prepareServer(CQLTester.java:360)
at org.apache.cassandra.cql3.CQLTester.setUpClass(CQLTester.java:411)
at org.apache.cassandra.cql3.CQLTester$InMemory.setUpClass(CQLTester.java:2876)
at org.apache.cassandra.repair.FuzzTestBase.setUpClass(FuzzTestBase.java:280)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
{code}
The reason this is happening is that it uses "InMemory.setUpClass" (5th line from the bottom). That does this:
{code}
@BeforeClass
public static void setUpClass()
{
fs = FileSystems.newGlobalInMemoryFileSystem();
CassandraRelevantProperties.IGNORE_MISSING_NATIVE_FILE_HINTS.setBoolean(true);
FileSystems.maybeCreateTmp();
CQLTester.setUpClass();
}
{code}
So, when it is "auto", previously, this failure was hidden, because commitlog dir was not created, hence "directIOSupported" was evaluated to false (the case described in the previous comment), so these test have probably ever run with "legacy" only. So once we want to enable that, it does not play together, because here "FileUtils.getBlockSize" fails with UnsupportedOperationException because it is most probably not supported in "FileSystems.newGlobalInMemoryFileSystem();".
{code}
public DirectIOSegmentBuilder(AbstractCommitLogSegmentManager segmentManager)
{
this(segmentManager, FileUtils.getBlockSize(new File(segmentManager.storageDirectory)));
}
{code}
I am trying to figure out what to do. This also means that we have never fuzz tested "direct" mode for cassandra_latest.yaml because it the missing commitlog dir never triggered it.
> direct IO support is always evaluated to false upon the very first start of a node
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-19779
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19779
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Tools
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
> Fix For: 5.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When I extract the distribution tarball and I want to use tools in tools/bin, there is this warn log visible every time for tools when they are started (does not happen on "help" command, obviously)
> {code:java}
> WARN 14:25:11,835 Unable to determine block size for commit log directory: null {code}
> This is because we introduced this (1) in CASSANDRA-18464
> What that does is that it will go and try to create a temporary file in commit log directory to get "block size" for a "file store" that file is in.
> The problem with that is that when we just extract a tarball and run the tools - Cassandra was never started - then such commit log directory does not exist yet, so it tries to create a temporary file in a non-existing directory, which fails, hence the log message.
> The fix is to check if commitlog dir exists and return / skip the resolution of block size if it does not.
> Another approach might be to check if this is executed in the context of a tool and skip it from resolution altogether. The problem with this is that not all tools we have in bin/log call DatabaseDescriptor.
> toolInitialization() so we might combine these two.
> (1) [https://github.com/apache/cassandra/blob/cassandra-5.0/src/java/org/apache/cassandra/config/DatabaseDescriptor.java#L1455-L1462]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org