You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flume.apache.org by mahendran m <ma...@hotmail.com> on 2014/11/18 06:51:29 UTC
Need help on spooling directory source
Hi All,
I was working on moving the IIS logs into HDFS using flume spooling directory sourcei was running the 3 website in IIS server , so my logging directory will looks like below
LogFiles|
|-- W3SVC1| || `--website1.log||-- W3SVC1| || `--website2.log|`-- W3SCV3
|
`-- website3.log
Now my use case is that i want to move the log of website 3 that is website3.log file to HDFS. So i have configured the CONF file as below
agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3
and it is works perfectly . now My use case is changed that i want move all the log files under the folder LogFiles , now i changed the CONF as below
agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles
but it does not working . not moving any files from LogFiles directory. How can i achieve my use case ?
Thanks,Mahendran
RE: Need help on spooling directory source
Posted by mahendran m <ma...@hotmail.com>.
Hi Manohar ,
I am using the flume version 1.5.0.1
Thanks,Mahendran
> From: Manohar.CS@itcinfotech.com
> To: user@flume.apache.org
> Subject: RE: Need help on spooling directory source
> Date: Tue, 18 Nov 2014 06:05:19 +0000
>
> Yes Ashish is right. Which flume version your are using?
>
> Version 1.4.0 does not support recursive directory checks, so, if you store your logs in a directory with a timestamp (i.e. /logs/project_name/2013/12/26), you will have to move all files to a single directory (i.e. /logs/project_name).
>
>
>
> -----Original Message-----
> From: Ashish [mailto:paliwalashish@gmail.com]
> Sent: Tuesday, November 18, 2014 11:32 AM
> To: user@flume.apache.org
> Subject: Re: Need help on spooling directory source
>
> AFAIK, it doesn't work recursively. It would pick files only from specified directory.
>
> Here is the patch you can try https://issues.apache.org/jira/browse/FLUME-1899
>
> HTH !
>
>
> On Tue, Nov 18, 2014 at 11:21 AM, mahendran m <ma...@hotmail.com> wrote:
> > Hi All,
> >
> > I was working on moving the IIS logs into HDFS using flume spooling
> > directory source i was running the 3 website in IIS server , so my
> > logging directory will looks like below
> >
> > LogFiles
> >
> > |
> > |-- W3SVC1
> >
> > | |
> >
> > | `--website1.log
> >
> > |
> >
> > |-- W3SVC1
> >
> > | |
> >
> > | `--website2.log
> >
> > |
> >
> > `-- W3SCV3
> > |
> > `-- website3.log
> >
> >
> >
> > Now my use case is that i want to move the log of website 3 that is
> > website3.log file to HDFS. So i have configured the CONF file as
> > below
> >
> >
> >
> > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3
> >
> >
> > and it is works perfectly . now My use case is changed that i want
> > move all the log files under the folder LogFiles , now i changed the
> > CONF as below
> >
> >
> > agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles
> >
> >
> > but it does not working . not moving any files from LogFiles
> > directory. How can i achieve my use case ?
> >
> >
> > Thanks,
> >
> > Mahendran
> >
> >
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>
> Please consider the environment before printing this e-mail
>
> Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
RE: Need help on spooling directory source
Posted by Manohar CS <Ma...@itcinfotech.com>.
Yes Ashish is right. Which flume version your are using?
Version 1.4.0 does not support recursive directory checks, so, if you store your logs in a directory with a timestamp (i.e. /logs/project_name/2013/12/26), you will have to move all files to a single directory (i.e. /logs/project_name).
-----Original Message-----
From: Ashish [mailto:paliwalashish@gmail.com]
Sent: Tuesday, November 18, 2014 11:32 AM
To: user@flume.apache.org
Subject: Re: Need help on spooling directory source
AFAIK, it doesn't work recursively. It would pick files only from specified directory.
Here is the patch you can try https://issues.apache.org/jira/browse/FLUME-1899
HTH !
On Tue, Nov 18, 2014 at 11:21 AM, mahendran m <ma...@hotmail.com> wrote:
> Hi All,
>
> I was working on moving the IIS logs into HDFS using flume spooling
> directory source i was running the 3 website in IIS server , so my
> logging directory will looks like below
>
> LogFiles
>
> |
> |-- W3SVC1
>
> | |
>
> | `--website1.log
>
> |
>
> |-- W3SVC1
>
> | |
>
> | `--website2.log
>
> |
>
> `-- W3SCV3
> |
> `-- website3.log
>
>
>
> Now my use case is that i want to move the log of website 3 that is
> website3.log file to HDFS. So i have configured the CONF file as
> below
>
>
>
> agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3
>
>
> and it is works perfectly . now My use case is changed that i want
> move all the log files under the folder LogFiles , now i changed the
> CONF as below
>
>
> agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles
>
>
> but it does not working . not moving any files from LogFiles
> directory. How can i achieve my use case ?
>
>
> Thanks,
>
> Mahendran
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Please consider the environment before printing this e-mail
Disclaimer: This communication is for the exclusive use of the intended recipient(s) and shall not attach any liability on the originator or ITC Infotech India Ltd./its Holding company/ its Subsidiaries/ its Group Companies. If you are the addressee, the contents of this e-mail are intended for your use only and it shall not be forwarded to any third party, without first obtaining written authorization from the originator or ITC Infotech India Ltd./ its Holding company/its Subsidiaries/ its Group Companies. It may contain information which is confidential and legally privileged and the same shall not be used or dealt with by any third party in any manner whatsoever without the specific consent of ITC Infotech India Ltd./ its Holding company/ its Subsidiaries/ its Group Companies.
Re: Need help on spooling directory source
Posted by Ashish <pa...@gmail.com>.
AFAIK, it doesn't work recursively. It would pick files only from
specified directory.
Here is the patch you can try https://issues.apache.org/jira/browse/FLUME-1899
HTH !
On Tue, Nov 18, 2014 at 11:21 AM, mahendran m <ma...@hotmail.com> wrote:
> Hi All,
>
> I was working on moving the IIS logs into HDFS using flume spooling
> directory source
> i was running the 3 website in IIS server , so my logging directory will
> looks like below
>
> LogFiles
>
> |
> |-- W3SVC1
>
> | |
>
> | `--website1.log
>
> |
>
> |-- W3SVC1
>
> | |
>
> | `--website2.log
>
> |
>
> `-- W3SCV3
> |
> `-- website3.log
>
>
>
> Now my use case is that i want to move the log of website 3 that is
> website3.log file to HDFS. So i have configured the CONF file as below
>
>
>
> agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles\\W3SVC3
>
>
> and it is works perfectly . now My use case is changed that i want move all
> the log files under the folder LogFiles , now i changed the CONF as below
>
>
> agent1.sources.spool.spoolDir =C:\\inetpub\\logs\\LogFiles
>
>
> but it does not working . not moving any files from LogFiles directory. How
> can i achieve my use case ?
>
>
> Thanks,
>
> Mahendran
>
>
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal