You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sachin Mittal <sj...@gmail.com> on 2019/12/01 07:22:04 UTC
Deleting topics in windows - fix estimate or workaround
Hi All,
I hope we are well aware of the critical bug in windows where kafka crashes
when we delete a topic. This affects other areas too like in stream
processing when trying to reset a stream using StreamsResetter.
Right now only workaround I have found is to stop zookeeper and kafka
server and manually delete directories and files containing streams, topics
and offset information.
However doing this every time is kind of unproductive.
I see that there are multiple critical bugs logged in JIRA
https://issues.apache.org/jira/browse/KAFKA-6203
https://issues.apache.org/jira/browse/KAFKA-1194
around the same issue.
I would like to know by when would the fix be available?
I see that there have been multiple pull requests issued around fixes of
these issues.
I wanted to know if one or more pull requests need to be merged to get the
fix out or if there is something I can try config wise to have some
workaround for this issue.
Please note that there can be few of us who might be using windows in
production too, so this fix is highly important for us.
Please let me know what can be done to address this issue.
Thanks
Sachin
Re: Deleting topics in windows - fix estimate or workaround
Posted by Sachin Mittal <sj...@gmail.com>.
Hi Edward,
I hear you.
So basically we are saying that unless major sponsors of Kafka project get
actual requirement for support on windows this issue will linger on.
Anyway my couple of takes from this issue is:
1. One should not overlook the potential opportunity for windows based
systems.
There may not be much incentive now, but if the software runs seamlessly on
windows, chances for such an opportunity increases.
2. There have been pull requests issued for fix of these issues, question
is how can we prioritize to get them to be merged in order to have a stable
windows release.
Thanks
Sachin
On Mon, Dec 2, 2019 at 5:34 AM Edward Capriolo <ed...@gmail.com>
wrote:
> My development environment is windows. I am actually able to run
> integration tests launching Kafka and Zookeeper. However I have noticed
> that the test warn likely about deleting a file that is locked and I tried
> deleting a topic at one point and saw some of the issues you mentioned. I
> personally think it is a blessing that it works at all, (because my
> environment does not support docker either so I would really be up the
> creek), but because it seems spotty in terms of features I do not count on
> it 100%.
>
> Because Java is supposedly 100% pure it seems like everything SHOULD work.
>
> I pessimistic hot take: Generally what (windows support) takes is some
> large organization (say the us government, or some bank) pays the largest
> consulting group/support company for a contract and demand windows support.
> Then magically it happens :)
>
> On Sun, Dec 1, 2019 at 12:47 PM Sachin Mittal <sj...@gmail.com> wrote:
>
> > Hi,
> > Firstly development environment is windows and I have been able to run
> > almost any big data framework on windows (or cygwin) be it HBase,
> Cassandra
> > or anything else.
> > Now just for running Kafka switching to linux is bit difficult.
> >
> > Further Kafka site does mention that windows is supported and I feel this
> > critical level bug should be addressed before making a release.
> > I clearly remember lower versions of Kafka did run on windows without any
> > problem. This issue started from 0.11 version I guess.
> >
> > Mostly in production we do shift to linux based OS or typically deploy on
> > such a cloud server.
> > However in this present case the client has a windows operating system
> and
> > we need to get this working without burdening them for linux based
> server.
> >
> > IT really moves slow in India and the time it will take them to provision
> > for a linux machines, the project may very well be lost.
> >
> > Would like to know if there is anything that can be done to make Kafka
> work
> > on windows.
> > I do feel that Kafka should continue to support windows because there are
> > still lot of businesses that use windows for some purpose where you do
> need
> > some kind of big data analysis.
> >
> > Like in this case we are trying to deploy a streams application in store
> > for collecting data from POS terminals which all run on windows.
> > All data analysis is store specific so we need a central server which can
> > process data from multiple terminals. Having them provision a windows
> > machine is very easy as that's all they have.
> >
> > Now to force them to install linux and that to over 1000's of stores
> nation
> > wide really needs some heavy lifting.
> >
> > Further moving the processing process to a central linux cloud is not
> > possible for now, because of lack of internet connectivity or a bad
> network
> > backbone, plus there are many more restrictions on moving data to cloud.
> >
> > So to conclude having a good windows support is essential for a thriving
> > kafka community and also to businesses that use and provide support on
> > kafka.
> >
> >
> > Please let me know how hard it would be to have a deployable kafka
> release
> > to windows. If there is some patch that can be released I am all good for
> > that too.
> >
> > Thanks
> > Sachin
> >
> >
> >
> > On Sun, Dec 1, 2019 at 8:54 PM Israel Ekpo <is...@gmail.com> wrote:
> >
> > > Sachin
> > >
> > > Are you running production Kafka on Windows?
> > >
> > > Is it possible to migrate to an alternate UNIX based environment?
> > >
> > > It would be great to learn why you chose to run it on Windows
> > >
> > >
> > > On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal <sj...@gmail.com>
> wrote:
> > >
> > > > Hi All,
> > > > I hope we are well aware of the critical bug in windows where kafka
> > > crashes
> > > > when we delete a topic. This affects other areas too like in stream
> > > > processing when trying to reset a stream using StreamsResetter.
> > > >
> > > > Right now only workaround I have found is to stop zookeeper and kafka
> > > > server and manually delete directories and files containing streams,
> > > topics
> > > > and offset information.
> > > >
> > > > However doing this every time is kind of unproductive.
> > > >
> > > > I see that there are multiple critical bugs logged in JIRA
> > > > https://issues.apache.org/jira/browse/KAFKA-6203
> > > > https://issues.apache.org/jira/browse/KAFKA-1194
> > > > around the same issue.
> > > >
> > > > I would like to know by when would the fix be available?
> > > > I see that there have been multiple pull requests issued around fixes
> > of
> > > > these issues.
> > > >
> > > > I wanted to know if one or more pull requests need to be merged to
> get
> > > the
> > > > fix out or if there is something I can try config wise to have some
> > > > workaround for this issue.
> > > >
> > > > Please note that there can be few of us who might be using windows in
> > > > production too, so this fix is highly important for us.
> > > >
> > > > Please let me know what can be done to address this issue.
> > > >
> > > > Thanks
> > > > Sachin
> > > >
> > >
> >
>
Re: Deleting topics in windows - fix estimate or workaround
Posted by Edward Capriolo <ed...@gmail.com>.
My development environment is windows. I am actually able to run
integration tests launching Kafka and Zookeeper. However I have noticed
that the test warn likely about deleting a file that is locked and I tried
deleting a topic at one point and saw some of the issues you mentioned. I
personally think it is a blessing that it works at all, (because my
environment does not support docker either so I would really be up the
creek), but because it seems spotty in terms of features I do not count on
it 100%.
Because Java is supposedly 100% pure it seems like everything SHOULD work.
I pessimistic hot take: Generally what (windows support) takes is some
large organization (say the us government, or some bank) pays the largest
consulting group/support company for a contract and demand windows support.
Then magically it happens :)
On Sun, Dec 1, 2019 at 12:47 PM Sachin Mittal <sj...@gmail.com> wrote:
> Hi,
> Firstly development environment is windows and I have been able to run
> almost any big data framework on windows (or cygwin) be it HBase, Cassandra
> or anything else.
> Now just for running Kafka switching to linux is bit difficult.
>
> Further Kafka site does mention that windows is supported and I feel this
> critical level bug should be addressed before making a release.
> I clearly remember lower versions of Kafka did run on windows without any
> problem. This issue started from 0.11 version I guess.
>
> Mostly in production we do shift to linux based OS or typically deploy on
> such a cloud server.
> However in this present case the client has a windows operating system and
> we need to get this working without burdening them for linux based server.
>
> IT really moves slow in India and the time it will take them to provision
> for a linux machines, the project may very well be lost.
>
> Would like to know if there is anything that can be done to make Kafka work
> on windows.
> I do feel that Kafka should continue to support windows because there are
> still lot of businesses that use windows for some purpose where you do need
> some kind of big data analysis.
>
> Like in this case we are trying to deploy a streams application in store
> for collecting data from POS terminals which all run on windows.
> All data analysis is store specific so we need a central server which can
> process data from multiple terminals. Having them provision a windows
> machine is very easy as that's all they have.
>
> Now to force them to install linux and that to over 1000's of stores nation
> wide really needs some heavy lifting.
>
> Further moving the processing process to a central linux cloud is not
> possible for now, because of lack of internet connectivity or a bad network
> backbone, plus there are many more restrictions on moving data to cloud.
>
> So to conclude having a good windows support is essential for a thriving
> kafka community and also to businesses that use and provide support on
> kafka.
>
>
> Please let me know how hard it would be to have a deployable kafka release
> to windows. If there is some patch that can be released I am all good for
> that too.
>
> Thanks
> Sachin
>
>
>
> On Sun, Dec 1, 2019 at 8:54 PM Israel Ekpo <is...@gmail.com> wrote:
>
> > Sachin
> >
> > Are you running production Kafka on Windows?
> >
> > Is it possible to migrate to an alternate UNIX based environment?
> >
> > It would be great to learn why you chose to run it on Windows
> >
> >
> > On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal <sj...@gmail.com> wrote:
> >
> > > Hi All,
> > > I hope we are well aware of the critical bug in windows where kafka
> > crashes
> > > when we delete a topic. This affects other areas too like in stream
> > > processing when trying to reset a stream using StreamsResetter.
> > >
> > > Right now only workaround I have found is to stop zookeeper and kafka
> > > server and manually delete directories and files containing streams,
> > topics
> > > and offset information.
> > >
> > > However doing this every time is kind of unproductive.
> > >
> > > I see that there are multiple critical bugs logged in JIRA
> > > https://issues.apache.org/jira/browse/KAFKA-6203
> > > https://issues.apache.org/jira/browse/KAFKA-1194
> > > around the same issue.
> > >
> > > I would like to know by when would the fix be available?
> > > I see that there have been multiple pull requests issued around fixes
> of
> > > these issues.
> > >
> > > I wanted to know if one or more pull requests need to be merged to get
> > the
> > > fix out or if there is something I can try config wise to have some
> > > workaround for this issue.
> > >
> > > Please note that there can be few of us who might be using windows in
> > > production too, so this fix is highly important for us.
> > >
> > > Please let me know what can be done to address this issue.
> > >
> > > Thanks
> > > Sachin
> > >
> >
>
Re: Deleting topics in windows - fix estimate or workaround
Posted by Sachin Mittal <sj...@gmail.com>.
Hi,
Firstly development environment is windows and I have been able to run
almost any big data framework on windows (or cygwin) be it HBase, Cassandra
or anything else.
Now just for running Kafka switching to linux is bit difficult.
Further Kafka site does mention that windows is supported and I feel this
critical level bug should be addressed before making a release.
I clearly remember lower versions of Kafka did run on windows without any
problem. This issue started from 0.11 version I guess.
Mostly in production we do shift to linux based OS or typically deploy on
such a cloud server.
However in this present case the client has a windows operating system and
we need to get this working without burdening them for linux based server.
IT really moves slow in India and the time it will take them to provision
for a linux machines, the project may very well be lost.
Would like to know if there is anything that can be done to make Kafka work
on windows.
I do feel that Kafka should continue to support windows because there are
still lot of businesses that use windows for some purpose where you do need
some kind of big data analysis.
Like in this case we are trying to deploy a streams application in store
for collecting data from POS terminals which all run on windows.
All data analysis is store specific so we need a central server which can
process data from multiple terminals. Having them provision a windows
machine is very easy as that's all they have.
Now to force them to install linux and that to over 1000's of stores nation
wide really needs some heavy lifting.
Further moving the processing process to a central linux cloud is not
possible for now, because of lack of internet connectivity or a bad network
backbone, plus there are many more restrictions on moving data to cloud.
So to conclude having a good windows support is essential for a thriving
kafka community and also to businesses that use and provide support on
kafka.
Please let me know how hard it would be to have a deployable kafka release
to windows. If there is some patch that can be released I am all good for
that too.
Thanks
Sachin
On Sun, Dec 1, 2019 at 8:54 PM Israel Ekpo <is...@gmail.com> wrote:
> Sachin
>
> Are you running production Kafka on Windows?
>
> Is it possible to migrate to an alternate UNIX based environment?
>
> It would be great to learn why you chose to run it on Windows
>
>
> On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal <sj...@gmail.com> wrote:
>
> > Hi All,
> > I hope we are well aware of the critical bug in windows where kafka
> crashes
> > when we delete a topic. This affects other areas too like in stream
> > processing when trying to reset a stream using StreamsResetter.
> >
> > Right now only workaround I have found is to stop zookeeper and kafka
> > server and manually delete directories and files containing streams,
> topics
> > and offset information.
> >
> > However doing this every time is kind of unproductive.
> >
> > I see that there are multiple critical bugs logged in JIRA
> > https://issues.apache.org/jira/browse/KAFKA-6203
> > https://issues.apache.org/jira/browse/KAFKA-1194
> > around the same issue.
> >
> > I would like to know by when would the fix be available?
> > I see that there have been multiple pull requests issued around fixes of
> > these issues.
> >
> > I wanted to know if one or more pull requests need to be merged to get
> the
> > fix out or if there is something I can try config wise to have some
> > workaround for this issue.
> >
> > Please note that there can be few of us who might be using windows in
> > production too, so this fix is highly important for us.
> >
> > Please let me know what can be done to address this issue.
> >
> > Thanks
> > Sachin
> >
>
Re: Deleting topics in windows - fix estimate or workaround
Posted by "M. Manna" <ma...@gmail.com>.
Hi both,
It’s been going around for a long time, but Kafka is officially not fully
tested and verified for Windows. The disclaimer is on the official site.
Windows servers are easy choice because a lot of infrastructures are on
Windows and a lot of businesses are dependent on those infrastructure. That
said, the forward way is highly likely to be on the cloud using some
orchestrator (Cloud Foundry) or IaaS provider e.g. AWS.
Also, Windows OS has proprietary settings on file systems which is the root
cause of this problem, so Kafka is out of scope for that. I would suggest
that you move to a Linux or UNIX like architecture (if at all possible)
Regards,
On Sun, 1 Dec 2019 at 15:24, Israel Ekpo <is...@gmail.com> wrote:
> Sachin
>
> Are you running production Kafka on Windows?
>
> Is it possible to migrate to an alternate UNIX based environment?
>
> It would be great to learn why you chose to run it on Windows
>
>
> On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal <sj...@gmail.com> wrote:
>
> > Hi All,
> > I hope we are well aware of the critical bug in windows where kafka
> crashes
> > when we delete a topic. This affects other areas too like in stream
> > processing when trying to reset a stream using StreamsResetter.
> >
> > Right now only workaround I have found is to stop zookeeper and kafka
> > server and manually delete directories and files containing streams,
> topics
> > and offset information.
> >
> > However doing this every time is kind of unproductive.
> >
> > I see that there are multiple critical bugs logged in JIRA
> > https://issues.apache.org/jira/browse/KAFKA-6203
> > https://issues.apache.org/jira/browse/KAFKA-1194
> > around the same issue.
> >
> > I would like to know by when would the fix be available?
> > I see that there have been multiple pull requests issued around fixes of
> > these issues.
> >
> > I wanted to know if one or more pull requests need to be merged to get
> the
> > fix out or if there is something I can try config wise to have some
> > workaround for this issue.
> >
> > Please note that there can be few of us who might be using windows in
> > production too, so this fix is highly important for us.
> >
> > Please let me know what can be done to address this issue.
> >
> > Thanks
> > Sachin
> >
>
Re: Deleting topics in windows - fix estimate or workaround
Posted by Israel Ekpo <is...@gmail.com>.
Sachin
Are you running production Kafka on Windows?
Is it possible to migrate to an alternate UNIX based environment?
It would be great to learn why you chose to run it on Windows
On Sun, Dec 1, 2019 at 2:22 AM Sachin Mittal <sj...@gmail.com> wrote:
> Hi All,
> I hope we are well aware of the critical bug in windows where kafka crashes
> when we delete a topic. This affects other areas too like in stream
> processing when trying to reset a stream using StreamsResetter.
>
> Right now only workaround I have found is to stop zookeeper and kafka
> server and manually delete directories and files containing streams, topics
> and offset information.
>
> However doing this every time is kind of unproductive.
>
> I see that there are multiple critical bugs logged in JIRA
> https://issues.apache.org/jira/browse/KAFKA-6203
> https://issues.apache.org/jira/browse/KAFKA-1194
> around the same issue.
>
> I would like to know by when would the fix be available?
> I see that there have been multiple pull requests issued around fixes of
> these issues.
>
> I wanted to know if one or more pull requests need to be merged to get the
> fix out or if there is something I can try config wise to have some
> workaround for this issue.
>
> Please note that there can be few of us who might be using windows in
> production too, so this fix is highly important for us.
>
> Please let me know what can be done to address this issue.
>
> Thanks
> Sachin
>