You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Phil Scala <Ph...@globalrelay.net> on 2013/04/13 02:07:51 UTC
[winows] tracker file sharing voilation wne trying to delete
tracker file
Hi,
I am new to Flume, was trying out some basic's on my Windows 7 dev machine and I ran into problems with the spooling directory source, the issue was in the org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile() method... we were getting an instance of a DurablePositionTracker and not calling close on it before then creating another instance... thus not calling close on the underlying DataFileWriter. This left the file locked in Windows and hence any delete attempts were failing...causing the Agent to spin around and around trying to delete this file and creating endless numbers of the temporary tracker files.
I have the code change here and can submit a JIRA item, I don't see anything similar in JIRA, but before I created something I wanted to discuss this in the community (realizing that Windows is not supported)
My code change is very simple, I added the close() call in the else block :
org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile()
....
if (!tracker.getTarget().equals(nextPath)) {
tracker.close();
deleteMetaFile();
}
else{
//Explicitly close the tracker instance received so that
//there is no sharing violation on Windows.
tracker.close();
}
tracker = DurablePositionTracker.getInstance(metaFile, nextPath);
....
Thanks
Phil
Phil Scala
Developer / Architect
Global Relay
phil.scala@globalrelay.net<ma...@globalrelay.net>
866.484.6630
New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore (+65.3158.1301)
Global Relay Archive supports email, instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, Facebook and more.
Ask about Global Relay Message<http://www.globalrelay.com/services/message> - The Future of Collaboration in the Financial Services World
All email sent to or from this address will be retained by Global Relay's email archiving system. This message is intended only for the use of the individual or entity to which it is addressed, and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. Global Relay will not be liable for any compliance or technical information provided herein. All trademarks are the property of their respective owners.
RE: [winows] tracker file sharing voilation wne trying to delete
tracker file
Posted by Phil Scala <Ph...@globalrelay.net>.
Thanks Hari, will do.
Phil
Phil Scala
Developer / Architect
Global Relay
phil.scala@globalrelay.net
866.484.6630 | info@globalrelay.net | globalrelay.com
-----Original Message-----
From: Hari Shreedharan [mailto:hshreedharan@cloudera.com]
Sent: April-12-13 5:20 PM
To: dev@flume.apache.org
Subject: Re: [winows] tracker file sharing voilation wne trying to delete tracker file
Hi Phil,
Thank you for reporting this. I believe Paul Chavez reported the same issue earlier today on the user@ list. Please file a jira and attach the patch.
Someone will review and commit it!
Thanks,
Hari
On Fri, Apr 12, 2013 at 5:07 PM, Phil Scala <Ph...@globalrelay.net>wrote:
> Hi,
>
> I am new to Flume, was trying out some basic's on my Windows 7 dev
> machine and I ran into problems with the spooling directory source,
> the issue was in the
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFi
> le() method... we were getting an instance of a DurablePositionTracker
> and not calling close on it before then creating another instance...
> thus not calling close on the underlying DataFileWriter. This left
> the file locked in Windows and hence any delete attempts were
> failing...causing the Agent to spin around and around trying to delete
> this file and creating endless numbers of the temporary tracker files.
>
> I have the code change here and can submit a JIRA item, I don't see
> anything similar in JIRA, but before I created something I wanted to
> discuss this in the community (realizing that Windows is not
> supported)
>
> My code change is very simple, I added the close() call in the else block :
>
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFi
> le()
> ....
>
> if (!tracker.getTarget().equals(nextPath)) {
> tracker.close();
> deleteMetaFile();
> }
> else{
> //Explicitly close the tracker instance received so that
> //there is no sharing violation on Windows.
> tracker.close();
> }
> tracker = DurablePositionTracker.getInstance(metaFile,
> nextPath); ....
>
> Thanks
> Phil
>
> Phil Scala
> Developer / Architect
> Global Relay
>
> phil.scala@globalrelay.net<ma...@globalrelay.net>
>
> 866.484.6630
> New York | Chicago | Vancouver | London (+44.0800.032.9829) |
> Singapore (+65.3158.1301) Global Relay Archive supports email,
> instant messaging, BlackBerry, Bloomberg, Thomson Reuters, Pivot,
> YellowJacket, LinkedIn, Twitter, Facebook and more.
>
> Ask about Global Relay
> Message<http://www.globalrelay.com/services/message>
> - The Future of Collaboration in the Financial Services World
>
> All email sent to or from this address will be retained by Global
> Relay's email archiving system. This message is intended only for the
> use of the individual or entity to which it is addressed, and may
> contain information that is privileged, confidential, and exempt from
> disclosure under applicable law. Global Relay will not be liable for
> any compliance or technical information provided herein. All
> trademarks are the property of their respective owners.
>
>
Re: [winows] tracker file sharing voilation wne trying to delete
tracker file
Posted by Hari Shreedharan <hs...@cloudera.com>.
Hi Phil,
Thank you for reporting this. I believe Paul Chavez reported the same issue
earlier today on the user@ list. Please file a jira and attach the patch.
Someone will review and commit it!
Thanks,
Hari
On Fri, Apr 12, 2013 at 5:07 PM, Phil Scala <Ph...@globalrelay.net>wrote:
> Hi,
>
> I am new to Flume, was trying out some basic's on my Windows 7 dev machine
> and I ran into problems with the spooling directory source, the issue was
> in the
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile()
> method... we were getting an instance of a DurablePositionTracker and not
> calling close on it before then creating another instance... thus not
> calling close on the underlying DataFileWriter. This left the file locked
> in Windows and hence any delete attempts were failing...causing the Agent
> to spin around and around trying to delete this file and creating endless
> numbers of the temporary tracker files.
>
> I have the code change here and can submit a JIRA item, I don't see
> anything similar in JIRA, but before I created something I wanted to
> discuss this in the community (realizing that Windows is not supported)
>
> My code change is very simple, I added the close() call in the else block :
>
> org.apache.flume.client.avro.ReliableSpoolingFileEventReader.getNextFile()
> ....
>
> if (!tracker.getTarget().equals(nextPath)) {
> tracker.close();
> deleteMetaFile();
> }
> else{
> //Explicitly close the tracker instance received so that
> //there is no sharing violation on Windows.
> tracker.close();
> }
> tracker = DurablePositionTracker.getInstance(metaFile, nextPath);
> ....
>
> Thanks
> Phil
>
> Phil Scala
> Developer / Architect
> Global Relay
>
> phil.scala@globalrelay.net<ma...@globalrelay.net>
>
> 866.484.6630
> New York | Chicago | Vancouver | London (+44.0800.032.9829) |
> Singapore (+65.3158.1301)
> Global Relay Archive supports email, instant messaging, BlackBerry,
> Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter,
> Facebook and more.
>
> Ask about Global Relay Message<http://www.globalrelay.com/services/message>
> - The Future of Collaboration in the Financial Services World
>
> All email sent to or from this address will be retained by Global Relay's
> email archiving system. This message is intended only for the use of the
> individual or entity to which it is addressed, and may contain information
> that is privileged, confidential, and exempt from disclosure under
> applicable law. Global Relay will not be liable for any compliance or
> technical information provided herein. All trademarks are the property of
> their respective owners.
>
>