You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Attila Wind <at...@swf.technology> on 2019/05/16 05:28:54 UTC
Running 4.0 (trunk) on Windows
Hi Devs,
Base question:
How to run Cassandra 4.0 (trunk) version on a Windows based environment
without getting always the errors like this?
org.apache.cassandra.io.FSReadError: java.io.IOException: Invalid folder
descriptor trying to create log replica
D:\workspace\java-projects\java-automated-test-suite\target\embeddedCassandra\data\test_keyspace\test_table-f6498e1075ca11e9a59ed3a612c57aaf
Background / Motivation:
In our project we want to use Map/List data types heavily - also
combined with IN operator. Due to
https://issues.apache.org/jira/browse/CASSANDRA-12654 this does not work
in 3.x but solved in 4.0. Also extended features around the above
complex data types in 4.0 excites me up. So I decided to do a test
deployment on our test servers (Linux based) - features worked like
charm there.
As a next step I extended our jBehave testing framework to also be able
to fire up an embedded Cassandra 4.0 for running tests. Since I am
developing on Windows I started to get the above mentioned FSReadError
exceptions however folder path is valid.
Same happens if I try to fire up Cassandra 4.0 instances on Windows systems.
More advanced question:
Based on stack traces I took a look into the source. In
org.apache.cassandra.db.lifecycle.LogReplica and e.g. in open(File file)
method (along with others) it is using the
NativeLibrary.tryOpenDirectory(file.getParentFile().getPath()) call - if
it fails I get the above FSReadError for sure.
And here we arrive to the main point... On Windows the
org.apache.cassandra.utils.NativeLibraryWindows class is used in the
background. But in this class basically almost all method implementation
is just "throw new UnsatisfiedLinkError();" - bingo! This shows me that
I have no really hope... If I am on Windows I will fail like hell with
above errors always.
1. Are you planning to do something around it?
What I did as an ass-saver I have added a "mock" implementation for 3
problematic methods by implementing in Java in NativeLibraryWindows
class fake_callOpen(), fake_callFsync(), fake_callClose() kinda
simulating something. fake_callOpen() stores the fake fdId => path
mapping in an internal Map object. fake_callClose() flushes that out.
fake_callFsync() gives a WARN if the fdId is uinknown (so
fake_callOpen() was not invoked earlier). Issue has gone for me after
this, Cassandra 4.0 runs fine with all of our test cases - I am happy!
:-) So.... and but...
2. My quick and dirty implementation shows that I regularly get lots of
warnings from my fake_callFsync(int fd): "file descriptor {} is unknown
so invalid!" which means callOpen(int fd) was NOT invoked in prior
calling callFsync() for the given descriptor. Is that normal??? Should
not callOpen() always be invoked??? Could it be a bug somehow???
3. I am really not planning ever run Cassandra (or any other) in Windows
based PROD system. But developing on Windows still. Wouldn't it make
sense to at least provide some workaround here for fellow developers
like me? At least to be able to run the server in Windows (for
testing/development purposes only) ...
thanks!
--
Attila Wind
http://www.linkedin.com/in/attilaw
Mobile: +36 31 7811355
Re: Running 4.0 (trunk) on Windows
Posted by Attila Wind <at...@swf.technology>.
Hi Joshua!
Thanks a lot for your quick answer - appreciated!
Makes sense 100% - also now I got my answers fully.
thanks
Attila
On 16.05.2019 2:58 PM, Joshua McKenzie wrote:
>
> there is no plan from Cassandra Devs to normally support Windows
> in Cassandra 4.0
>
> I was the dev kind of single-handedly making it work on Windows; it
> was good up through the 3.X line but I didn't maintain it past that
> point to 4.0+ and don't have plans to. Reason being: the WSL approach
> 100% works for dev environments, and while I/O is pretty slow right
> now (i.e. use for dev only) there's a new version of Windows hitting
> soon with an actual linux kernel embedded in it[1] where the I/O perf
> issue should be resolved to the point of it being viable to run linux
> native apps on WSL in production. So no real need for "windows
> support" on the project, per se, as Windows will have support *for it*. :)
>
> [1]https://www.techtimes.com/articles/242972/20190514/future-windows-10-versions-to-ship-with-real-linux-kernel-for-wsl.htm
>
> On Thu, May 16, 2019 at 2:26 AM Attila Wind <at...@swf.technology>
> wrote:
>
> Hi Dinesh,
>
> No. And I am not really planning to. Too early to start looking for
> other workarounds I think (I could also put the whole stuff into a
> Linux
> based Docker container and run from there if it gets to that point
> - its
> supported by our everything) before it is confirmed fact that
> there is
> no plan from Cassandra Devs to normally support Windows in
> Cassandra 4.0.
> Currently it is not the case as far as I see so for now more
> interested
> about intentions - that was the goal of my mail. Plus of course also
> give myself a chance to let it turn out if I am missing a config
> options
> somewhere somehow... :-) Code base is pretty big I absolutely have no
> overview of course. Chance is high I am missing something... :-)
>
> thanks
>
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 16. 7:49, Dinesh Joshi wrote:
> > Hi Attila,
> >
> > Have you tried building & running Cassandra using Windows
> Subsystem for Linux[1]? The integration is seamless and you get a
> full fledged Linux environment on Windows. The overhead is minimal
> and you should be able to edit in your IDE on Windows.
> >
> > Thanks,
> >
> > Dinesh
> >
> > [1] https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
> >
> >> On May 15, 2019, at 10:28 PM, Attila Wind
> <at...@swf.technology> wrote:
> >>
> >> Hi Devs,
> >>
> >> Base question:
> >> How to run Cassandra 4.0 (trunk) version on a Windows based
> environment without getting always the errors like this?
> >> org.apache.cassandra.io
> <http://org.apache.cassandra.io>.FSReadError: java.io.IOException:
> Invalid folder descriptor trying to create log replica
> D:\workspace\java-projects\java-automated-test-suite\target\embeddedCassandra\data\test_keyspace\test_table-f6498e1075ca11e9a59ed3a612c57aaf
> >>
> >> Background / Motivation:
> >> In our project we want to use Map/List data types heavily -
> also combined with IN operator. Due to
> https://issues.apache.org/jira/browse/CASSANDRA-12654 this does
> not work in 3.x but solved in 4.0. Also extended features around
> the above complex data types in 4.0 excites me up. So I decided to
> do a test deployment on our test servers (Linux based) - features
> worked like charm there.
> >> As a next step I extended our jBehave testing framework to also
> be able to fire up an embedded Cassandra 4.0 for running tests.
> Since I am developing on Windows I started to get the above
> mentioned FSReadError exceptions however folder path is valid.
> >> Same happens if I try to fire up Cassandra 4.0 instances on
> Windows systems.
> >>
> >> More advanced question:
> >> Based on stack traces I took a look into the source. In
> org.apache.cassandra.db.lifecycle.LogReplica and e.g. in open(File
> file) method (along with others) it is using the
> NativeLibrary.tryOpenDirectory(file.getParentFile().getPath())
> call - if it fails I get the above FSReadError for sure.
> >> And here we arrive to the main point... On Windows the
> org.apache.cassandra.utils.NativeLibraryWindows class is used in
> the background. But in this class basically almost all method
> implementation is just "throw new UnsatisfiedLinkError();" -
> bingo! This shows me that I have no really hope... If I am on
> Windows I will fail like hell with above errors always.
> >>
> >> 1. Are you planning to do something around it?
> >>
> >> What I did as an ass-saver I have added a "mock" implementation
> for 3 problematic methods by implementing in Java in
> NativeLibraryWindows class fake_callOpen(), fake_callFsync(),
> fake_callClose() kinda simulating something. fake_callOpen()
> stores the fake fdId => path mapping in an internal Map object.
> fake_callClose() flushes that out. fake_callFsync() gives a WARN
> if the fdId is uinknown (so fake_callOpen() was not invoked
> earlier). Issue has gone for me after this, Cassandra 4.0 runs
> fine with all of our test cases - I am happy! :-) So.... and but...
> >>
> >> 2. My quick and dirty implementation shows that I regularly get
> lots of warnings from my fake_callFsync(int fd): "file descriptor
> {} is unknown so invalid!" which means callOpen(int fd) was NOT
> invoked in prior calling callFsync() for the given descriptor. Is
> that normal??? Should not callOpen() always be invoked??? Could it
> be a bug somehow???
> >>
> >> 3. I am really not planning ever run Cassandra (or any other)
> in Windows based PROD system. But developing on Windows still.
> Wouldn't it make sense to at least provide some workaround here
> for fellow developers like me? At least to be able to run the
> server in Windows (for testing/development purposes only) ...
> >>
> >> thanks!
> >>
> >> --
> >> Attila Wind
> >>
> >> http://www.linkedin.com/in/attilaw
> >> Mobile: +36 31 7811355
> >>
> >>
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> <ma...@cassandra.apache.org>
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> <ma...@cassandra.apache.org>
> >
>
Re: Running 4.0 (trunk) on Windows
Posted by Joshua McKenzie <jm...@apache.org>.
>
> there is no plan from Cassandra Devs to normally support Windows in
> Cassandra 4.0
I was the dev kind of single-handedly making it work on Windows; it was
good up through the 3.X line but I didn't maintain it past that point to
4.0+ and don't have plans to. Reason being: the WSL approach 100% works for
dev environments, and while I/O is pretty slow right now (i.e. use for dev
only) there's a new version of Windows hitting soon with an actual linux
kernel embedded in it[1] where the I/O perf issue should be resolved to the
point of it being viable to run linux native apps on WSL in production. So
no real need for "windows support" on the project, per se, as Windows will
have support *for it*. :)
[1]
https://www.techtimes.com/articles/242972/20190514/future-windows-10-versions-to-ship-with-real-linux-kernel-for-wsl.htm
On Thu, May 16, 2019 at 2:26 AM Attila Wind <at...@swf.technology> wrote:
> Hi Dinesh,
>
> No. And I am not really planning to. Too early to start looking for
> other workarounds I think (I could also put the whole stuff into a Linux
> based Docker container and run from there if it gets to that point - its
> supported by our everything) before it is confirmed fact that there is
> no plan from Cassandra Devs to normally support Windows in Cassandra 4.0.
> Currently it is not the case as far as I see so for now more interested
> about intentions - that was the goal of my mail. Plus of course also
> give myself a chance to let it turn out if I am missing a config options
> somewhere somehow... :-) Code base is pretty big I absolutely have no
> overview of course. Chance is high I am missing something... :-)
>
> thanks
>
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
> On 2019. 05. 16. 7:49, Dinesh Joshi wrote:
> > Hi Attila,
> >
> > Have you tried building & running Cassandra using Windows Subsystem for
> Linux[1]? The integration is seamless and you get a full fledged Linux
> environment on Windows. The overhead is minimal and you should be able to
> edit in your IDE on Windows.
> >
> > Thanks,
> >
> > Dinesh
> >
> > [1] https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
> >
> >> On May 15, 2019, at 10:28 PM, Attila Wind <at...@swf.technology>
> wrote:
> >>
> >> Hi Devs,
> >>
> >> Base question:
> >> How to run Cassandra 4.0 (trunk) version on a Windows based environment
> without getting always the errors like this?
> >> org.apache.cassandra.io.FSReadError: java.io.IOException: Invalid
> folder descriptor trying to create log replica
> D:\workspace\java-projects\java-automated-test-suite\target\embeddedCassandra\data\test_keyspace\test_table-f6498e1075ca11e9a59ed3a612c57aaf
> >>
> >> Background / Motivation:
> >> In our project we want to use Map/List data types heavily - also
> combined with IN operator. Due to
> https://issues.apache.org/jira/browse/CASSANDRA-12654 this does not work
> in 3.x but solved in 4.0. Also extended features around the above complex
> data types in 4.0 excites me up. So I decided to do a test deployment on
> our test servers (Linux based) - features worked like charm there.
> >> As a next step I extended our jBehave testing framework to also be able
> to fire up an embedded Cassandra 4.0 for running tests. Since I am
> developing on Windows I started to get the above mentioned FSReadError
> exceptions however folder path is valid.
> >> Same happens if I try to fire up Cassandra 4.0 instances on Windows
> systems.
> >>
> >> More advanced question:
> >> Based on stack traces I took a look into the source. In
> org.apache.cassandra.db.lifecycle.LogReplica and e.g. in open(File file)
> method (along with others) it is using the
> NativeLibrary.tryOpenDirectory(file.getParentFile().getPath()) call - if it
> fails I get the above FSReadError for sure.
> >> And here we arrive to the main point... On Windows the
> org.apache.cassandra.utils.NativeLibraryWindows class is used in the
> background. But in this class basically almost all method implementation is
> just "throw new UnsatisfiedLinkError();" - bingo! This shows me that I have
> no really hope... If I am on Windows I will fail like hell with above
> errors always.
> >>
> >> 1. Are you planning to do something around it?
> >>
> >> What I did as an ass-saver I have added a "mock" implementation for 3
> problematic methods by implementing in Java in NativeLibraryWindows class
> fake_callOpen(), fake_callFsync(), fake_callClose() kinda simulating
> something. fake_callOpen() stores the fake fdId => path mapping in an
> internal Map object. fake_callClose() flushes that out. fake_callFsync()
> gives a WARN if the fdId is uinknown (so fake_callOpen() was not invoked
> earlier). Issue has gone for me after this, Cassandra 4.0 runs fine with
> all of our test cases - I am happy! :-) So.... and but...
> >>
> >> 2. My quick and dirty implementation shows that I regularly get lots of
> warnings from my fake_callFsync(int fd): "file descriptor {} is unknown so
> invalid!" which means callOpen(int fd) was NOT invoked in prior calling
> callFsync() for the given descriptor. Is that normal??? Should not
> callOpen() always be invoked??? Could it be a bug somehow???
> >>
> >> 3. I am really not planning ever run Cassandra (or any other) in
> Windows based PROD system. But developing on Windows still. Wouldn't it
> make sense to at least provide some workaround here for fellow developers
> like me? At least to be able to run the server in Windows (for
> testing/development purposes only) ...
> >>
> >> thanks!
> >>
> >> --
> >> Attila Wind
> >>
> >> http://www.linkedin.com/in/attilaw
> >> Mobile: +36 31 7811355
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
>
Re: Running 4.0 (trunk) on Windows
Posted by Attila Wind <at...@swf.technology>.
Hi Dinesh,
No. And I am not really planning to. Too early to start looking for
other workarounds I think (I could also put the whole stuff into a Linux
based Docker container and run from there if it gets to that point - its
supported by our everything) before it is confirmed fact that there is
no plan from Cassandra Devs to normally support Windows in Cassandra 4.0.
Currently it is not the case as far as I see so for now more interested
about intentions - that was the goal of my mail. Plus of course also
give myself a chance to let it turn out if I am missing a config options
somewhere somehow... :-) Code base is pretty big I absolutely have no
overview of course. Chance is high I am missing something... :-)
thanks
Attila Wind
http://www.linkedin.com/in/attilaw
Mobile: +36 31 7811355
On 2019. 05. 16. 7:49, Dinesh Joshi wrote:
> Hi Attila,
>
> Have you tried building & running Cassandra using Windows Subsystem for Linux[1]? The integration is seamless and you get a full fledged Linux environment on Windows. The overhead is minimal and you should be able to edit in your IDE on Windows.
>
> Thanks,
>
> Dinesh
>
> [1] https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
>
>> On May 15, 2019, at 10:28 PM, Attila Wind <at...@swf.technology> wrote:
>>
>> Hi Devs,
>>
>> Base question:
>> How to run Cassandra 4.0 (trunk) version on a Windows based environment without getting always the errors like this?
>> org.apache.cassandra.io.FSReadError: java.io.IOException: Invalid folder descriptor trying to create log replica D:\workspace\java-projects\java-automated-test-suite\target\embeddedCassandra\data\test_keyspace\test_table-f6498e1075ca11e9a59ed3a612c57aaf
>>
>> Background / Motivation:
>> In our project we want to use Map/List data types heavily - also combined with IN operator. Due to https://issues.apache.org/jira/browse/CASSANDRA-12654 this does not work in 3.x but solved in 4.0. Also extended features around the above complex data types in 4.0 excites me up. So I decided to do a test deployment on our test servers (Linux based) - features worked like charm there.
>> As a next step I extended our jBehave testing framework to also be able to fire up an embedded Cassandra 4.0 for running tests. Since I am developing on Windows I started to get the above mentioned FSReadError exceptions however folder path is valid.
>> Same happens if I try to fire up Cassandra 4.0 instances on Windows systems.
>>
>> More advanced question:
>> Based on stack traces I took a look into the source. In org.apache.cassandra.db.lifecycle.LogReplica and e.g. in open(File file) method (along with others) it is using the NativeLibrary.tryOpenDirectory(file.getParentFile().getPath()) call - if it fails I get the above FSReadError for sure.
>> And here we arrive to the main point... On Windows the org.apache.cassandra.utils.NativeLibraryWindows class is used in the background. But in this class basically almost all method implementation is just "throw new UnsatisfiedLinkError();" - bingo! This shows me that I have no really hope... If I am on Windows I will fail like hell with above errors always.
>>
>> 1. Are you planning to do something around it?
>>
>> What I did as an ass-saver I have added a "mock" implementation for 3 problematic methods by implementing in Java in NativeLibraryWindows class fake_callOpen(), fake_callFsync(), fake_callClose() kinda simulating something. fake_callOpen() stores the fake fdId => path mapping in an internal Map object. fake_callClose() flushes that out. fake_callFsync() gives a WARN if the fdId is uinknown (so fake_callOpen() was not invoked earlier). Issue has gone for me after this, Cassandra 4.0 runs fine with all of our test cases - I am happy! :-) So.... and but...
>>
>> 2. My quick and dirty implementation shows that I regularly get lots of warnings from my fake_callFsync(int fd): "file descriptor {} is unknown so invalid!" which means callOpen(int fd) was NOT invoked in prior calling callFsync() for the given descriptor. Is that normal??? Should not callOpen() always be invoked??? Could it be a bug somehow???
>>
>> 3. I am really not planning ever run Cassandra (or any other) in Windows based PROD system. But developing on Windows still. Wouldn't it make sense to at least provide some workaround here for fellow developers like me? At least to be able to run the server in Windows (for testing/development purposes only) ...
>>
>> thanks!
>>
>> --
>> Attila Wind
>>
>> http://www.linkedin.com/in/attilaw
>> Mobile: +36 31 7811355
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: dev-help@cassandra.apache.org
>
Re: Running 4.0 (trunk) on Windows
Posted by Dinesh Joshi <dj...@apache.org>.
Hi Attila,
Have you tried building & running Cassandra using Windows Subsystem for Linux[1]? The integration is seamless and you get a full fledged Linux environment on Windows. The overhead is minimal and you should be able to edit in your IDE on Windows.
Thanks,
Dinesh
[1] https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux
> On May 15, 2019, at 10:28 PM, Attila Wind <at...@swf.technology> wrote:
>
> Hi Devs,
>
> Base question:
> How to run Cassandra 4.0 (trunk) version on a Windows based environment without getting always the errors like this?
> org.apache.cassandra.io.FSReadError: java.io.IOException: Invalid folder descriptor trying to create log replica D:\workspace\java-projects\java-automated-test-suite\target\embeddedCassandra\data\test_keyspace\test_table-f6498e1075ca11e9a59ed3a612c57aaf
>
> Background / Motivation:
> In our project we want to use Map/List data types heavily - also combined with IN operator. Due to https://issues.apache.org/jira/browse/CASSANDRA-12654 this does not work in 3.x but solved in 4.0. Also extended features around the above complex data types in 4.0 excites me up. So I decided to do a test deployment on our test servers (Linux based) - features worked like charm there.
> As a next step I extended our jBehave testing framework to also be able to fire up an embedded Cassandra 4.0 for running tests. Since I am developing on Windows I started to get the above mentioned FSReadError exceptions however folder path is valid.
> Same happens if I try to fire up Cassandra 4.0 instances on Windows systems.
>
> More advanced question:
> Based on stack traces I took a look into the source. In org.apache.cassandra.db.lifecycle.LogReplica and e.g. in open(File file) method (along with others) it is using the NativeLibrary.tryOpenDirectory(file.getParentFile().getPath()) call - if it fails I get the above FSReadError for sure.
> And here we arrive to the main point... On Windows the org.apache.cassandra.utils.NativeLibraryWindows class is used in the background. But in this class basically almost all method implementation is just "throw new UnsatisfiedLinkError();" - bingo! This shows me that I have no really hope... If I am on Windows I will fail like hell with above errors always.
>
> 1. Are you planning to do something around it?
>
> What I did as an ass-saver I have added a "mock" implementation for 3 problematic methods by implementing in Java in NativeLibraryWindows class fake_callOpen(), fake_callFsync(), fake_callClose() kinda simulating something. fake_callOpen() stores the fake fdId => path mapping in an internal Map object. fake_callClose() flushes that out. fake_callFsync() gives a WARN if the fdId is uinknown (so fake_callOpen() was not invoked earlier). Issue has gone for me after this, Cassandra 4.0 runs fine with all of our test cases - I am happy! :-) So.... and but...
>
> 2. My quick and dirty implementation shows that I regularly get lots of warnings from my fake_callFsync(int fd): "file descriptor {} is unknown so invalid!" which means callOpen(int fd) was NOT invoked in prior calling callFsync() for the given descriptor. Is that normal??? Should not callOpen() always be invoked??? Could it be a bug somehow???
>
> 3. I am really not planning ever run Cassandra (or any other) in Windows based PROD system. But developing on Windows still. Wouldn't it make sense to at least provide some workaround here for fellow developers like me? At least to be able to run the server in Windows (for testing/development purposes only) ...
>
> thanks!
>
> --
> Attila Wind
>
> http://www.linkedin.com/in/attilaw
> Mobile: +36 31 7811355
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org