You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by "Mahadevappa, Shobha" <Sh...@nttdata.com> on 2014/12/03 08:37:02 UTC
Flume on windows
Hi,
Can anyone please let me know if flume ng is supported on windows?
If yes, please provide pointers to the same.
Regards,
Shobha M
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
Re: Flume on windows
Posted by Hang Yan <ha...@iflytek.com>.
We have many flume agent run on windows,but we use a modifed version of
file channel.
From: Frank Yao <ba...@gmail.com>
Reply-To: <us...@flume.apache.org>
Date: Thursday, December 4, 2014 at 09:13
To: "user@flume.apache.org" <us...@flume.apache.org>
Subject: Re: Flume on windows
We are facing the same problem. Should flume do some comments to tell if
this component is suitable for Windows?
Frank Yao
@Vipshop, Shanghai
from iPhone
在 2014年12月4日，08:19，Hari Shreedharan <hs...@cloudera.com> 写道：
Theoretically, it might work - but there are several flume components that
use fsync etc which won’t work on Windows. So any guarantees provided by
things like the File Channel are voided. That said, I have never tested it
on Windows - so I can’t promise anything will work.
Thanks,
Hari
On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha
<Sh...@nttdata.com> wrote:
Hi,
Can anyone please let me know if flume ng is supported on windows?
If yes, please provide pointers to the same.
Regards,
Shobha M
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
Re: Flume on windows
Posted by Frank Yao <ba...@gmail.com>.
We are facing the same problem. Should flume do some comments to tell if this component is suitable for Windows?
Frank Yao
@Vipshop, Shanghai
from iPhone
> 在 2014年12月4日，08:19，Hari Shreedharan <hs...@cloudera.com> 写道：
>
> Theoretically, it might work - but there are several flume components that use fsync etc which won’t work on Windows. So any guarantees provided by things like the File Channel are voided. That said, I have never tested it on Windows - so I can’t promise anything will work.
>
> Thanks,
> Hari
>
>
>> On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha <Sh...@nttdata.com> wrote:
>> Hi,
>>
>> Can anyone please let me know if flume ng is supported on windows?
>>
>> If yes, please provide pointers to the same.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Shobha M
>>
>>
>>
>>
>> ______________________________________________________________________
>> Disclaimer: This email and any attachments are sent in strictest confidence
>> for the sole use of the addressee and may contain legally privileged,
>> confidential, and proprietary data. If you are not the intended recipient,
>> please advise the sender by replying promptly to this email and then delete
>> and destroy this email and any attachments without any further use, copying
>> or forwarding.
>
RE: Flume on windows
Posted by "Mahadevappa, Shobha" <Sh...@nttdata.com>.
I tried this and it did work. Thanks.
Regards,
Shobha
From: j.guilmard@accenture.com [mailto:j.guilmard@accenture.com]
Sent: Thursday, December 04, 2014 7:17 PM
To: user@flume.apache.org
Subject: RE: Flume on windows
Hi,
I do not use the File options with Flume on Windows: only memory channel and external sources. Nevertheless, might be worth trying a tail implementation for Windows (like http://gnuwin32.sourceforge.net/packages/coreutils.htm ), and use the Exec Source ?
Jeff
From: Mahadevappa, Shobha [mailto:Shobha.Mahadevappa@nttdata.com]
Sent: jeudi 4 décembre 2014 11:40
To: user@flume.apache.org
Subject: RE: Flume on windows
Thanks all for the inputs.
I tried the latest flume version on windows and was successful in starting up the collector and agents.
However, one of the agent source tails a file which fails on windows.
Any inputs how this can be achieved?
Regards,
Shobha
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, December 04, 2014 5:49 AM
To: user@flume.apache.org<ma...@flume.apache.org>
Cc: user@flume.apache.org<ma...@flume.apache.org>
Subject: Re: Flume on windows
Theoretically, it might work - but there are several flume components that use fsync etc which won’t work on Windows. So any guarantees provided by things like the File Channel are voided. That said, I have never tested it on Windows - so I can’t promise anything will work.
Thanks,
Hari
On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha <Sh...@nttdata.com>> wrote:
Hi,
Can anyone please let me know if flume ng is supported on windows?
If yes, please provide pointers to the same.
Regards,
Shobha M
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com<http://www.accenture.com>
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
Re: Flume on windows
Posted by Roshan Naik <ro...@hortonworks.com>.
This topic keeps coming up every now and then. One way to get it working
on windows is just try to cross your fingers and run the plain vanilla
apache distro ... your mileage will vary significantly depending on what
components and features you use.
for eg. spool dir with the default deserializer wont work correctly due to
improper handling of new lines, dual checkpointing needs to be disabled on
windows, etc.
There are actually several windows patches currently waiting to be
committed for windows support. If you search for all Jiras with "windows"
in the title you should be able to find them. You can either apply the ones
relevant to your needs and create a custom build or apply them all.
One of the patches includes a startup script for windows. IMO that should
be useful as it can alleviate inconveniences related to establishing the
right environment for flume.
If you prefer to not build it yourself, you could consider using some
pre-packaged distros like HDP that have Windows support.
On Thu, Dec 4, 2014 at 5:47 AM, <j....@accenture.com> wrote:
> Hi,
>
>
>
> I do not use the File options with Flume on Windows: only memory channel
> and external sources. Nevertheless, might be worth trying a tail
> implementation for Windows (like
> http://gnuwin32.sourceforge.net/packages/coreutils.htm ), and use the
> Exec Source ?
>
>
>
> *Jeff*
>
>
>
> *From:* Mahadevappa, Shobha [mailto:Shobha.Mahadevappa@nttdata.com]
> *Sent:* jeudi 4 décembre 2014 11:40
> *To:* user@flume.apache.org
> *Subject:* RE: Flume on windows
>
>
>
> Thanks all for the inputs.
>
> I tried the latest flume version on windows and was successful in starting
> up the collector and agents.
>
> However, one of the agent source tails a file which fails on windows.
>
> Any inputs how this can be achieved?
>
>
>
> Regards,
>
> Shobha
>
>
>
> *From:* Hari Shreedharan [mailto:hshreedharan@cloudera.com
> <hs...@cloudera.com>]
> *Sent:* Thursday, December 04, 2014 5:49 AM
> *To:* user@flume.apache.org
> *Cc:* user@flume.apache.org
> *Subject:* Re: Flume on windows
>
>
>
> Theoretically, it might work - but there are several flume components that
> use fsync etc which won’t work on Windows. So any guarantees provided by
> things like the File Channel are voided. That said, I have never tested it
> on Windows - so I can’t promise anything will work.
>
>
> Thanks,
> Hari
>
>
>
> On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha <
> Shobha.Mahadevappa@nttdata.com> wrote:
>
> Hi,
>
> Can anyone please let me know if flume ng is supported on windows?
>
> If yes, please provide pointers to the same.
>
>
>
>
>
> Regards,
>
> *Shobha M *
>
>
>
>
> ______________________________________________________________________
> Disclaimer: This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding.
>
>
>
>
> ______________________________________________________________________
> Disclaimer: This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding.
>
> ------------------------------
>
> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise confidential information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited. Where allowed
> by local law, electronic communications with Accenture and its affiliates,
> including e-mail and instant messaging (including content), may be scanned
> by our systems for the purposes of information security and assessment of
> internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
RE: Flume on windows
Posted by j....@accenture.com.
Hi,
I do not use the File options with Flume on Windows: only memory channel and external sources. Nevertheless, might be worth trying a tail implementation for Windows (like http://gnuwin32.sourceforge.net/packages/coreutils.htm ), and use the Exec Source ?
Jeff
From: Mahadevappa, Shobha [mailto:Shobha.Mahadevappa@nttdata.com]
Sent: jeudi 4 décembre 2014 11:40
To: user@flume.apache.org
Subject: RE: Flume on windows
Thanks all for the inputs.
I tried the latest flume version on windows and was successful in starting up the collector and agents.
However, one of the agent source tails a file which fails on windows.
Any inputs how this can be achieved?
Regards,
Shobha
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, December 04, 2014 5:49 AM
To: user@flume.apache.org<ma...@flume.apache.org>
Cc: user@flume.apache.org<ma...@flume.apache.org>
Subject: Re: Flume on windows
Theoretically, it might work - but there are several flume components that use fsync etc which won’t work on Windows. So any guarantees provided by things like the File Channel are voided. That said, I have never tested it on Windows - so I can’t promise anything will work.
Thanks,
Hari
On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha <Sh...@nttdata.com>> wrote:
Hi,
Can anyone please let me know if flume ng is supported on windows?
If yes, please provide pointers to the same.
Regards,
Shobha M
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
________________________________
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com
RE: Flume on windows
Posted by "Mahadevappa, Shobha" <Sh...@nttdata.com>.
Thanks all for the inputs.
I tried the latest flume version on windows and was successful in starting up the collector and agents.
However, one of the agent source tails a file which fails on windows.
Any inputs how this can be achieved?
Regards,
Shobha
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: Thursday, December 04, 2014 5:49 AM
To: user@flume.apache.org
Cc: user@flume.apache.org
Subject: Re: Flume on windows
Theoretically, it might work - but there are several flume components that use fsync etc which won’t work on Windows. So any guarantees provided by things like the File Channel are voided. That said, I have never tested it on Windows - so I can’t promise anything will work.
Thanks,
Hari
On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha <Sh...@nttdata.com>> wrote:
Hi,
Can anyone please let me know if flume ng is supported on windows?
If yes, please provide pointers to the same.
Regards,
Shobha M
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
______________________________________________________________________
Disclaimer: This email and any attachments are sent in strictest confidence
for the sole use of the addressee and may contain legally privileged,
confidential, and proprietary data. If you are not the intended recipient,
please advise the sender by replying promptly to this email and then delete
and destroy this email and any attachments without any further use, copying
or forwarding.
Re: Flume on windows
Posted by Hari Shreedharan <hs...@cloudera.com>.
Theoretically, it might work - but there are several flume components that use fsync etc which won’t work on Windows. So any guarantees provided by things like the File Channel are voided. That said, I have never tested it on Windows - so I can’t promise anything will work.
Thanks,
Hari
On Wed, Dec 3, 2014 at 2:40 AM, Mahadevappa, Shobha
<Sh...@nttdata.com> wrote:
> Hi,
> Can anyone please let me know if flume ng is supported on windows?
> If yes, please provide pointers to the same.
> Regards,
> Shobha M
> ______________________________________________________________________
> Disclaimer: This email and any attachments are sent in strictest confidence
> for the sole use of the addressee and may contain legally privileged,
> confidential, and proprietary data. If you are not the intended recipient,
> please advise the sender by replying promptly to this email and then delete
> and destroy this email and any attachments without any further use, copying
> or forwarding.