Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On UNC Path

[Justin Crossman <ju...@deliriousvisions.com>]

I am very new to this list, thank you for indulging me.

I'm dealing with an issue and have been unable to find any posts or documentation that already address this specific problem.

The setup:

This issue is regarding a heavily accessed web site, written in .NET 3.5 and running on IIS7. The application utilizes a Lucene search index via UNC path set within the application's Web.config. This UNC path is the same on each of 16 web servers and they all access the same index in this single location. The application runs in classic mode (vs integrated) and it uses an identity and impersonation. The account has sufficient permissions. The web servers are all currently Windows 2008 servers and the source file share is also Windows 2008 server running SQL Server 2008 as well. The servers are always in a Workstation configuration, never a domain and no AD. All servers have one main user account configured identically with identical permissions and password.

The history:

This setup has been working fine for years under a few configurations. Recently with web servers running Windows 2003 and IIS6 accessing this remote UNC file share on a Windows 2003 server running SQL Server 2005. And most recently with web servers running Windows 2008 and IIS7 accessing this remote UNC file share on the same Windows 2003 with SQL as just referred to. The problem occurs only now after having moved this UNC share to this new Windows 2008 server with SQL.

The problem:

The site runs well currently and each web server has access to the index and successfully utilizes the index as it should. However, periodically (between 1 and 15 minutes based on activity and a timed service) the Lucene "segments" file is removed and replaced (it's changed). The current file, e.g. "segments_xyz" is replaced with "segments_zyx". Not every time this occurs, but most times, the application seeks out the previous file, not the current file. This results in a FileNotFoundException and the .NET error reported is: System.IO.FileNotFoundException: Could not find file '\X.X.X.X\Index\20101201\guid-x-x-x-x\segments_zyx'. This lasts for 1 to 3 seconds and it occurs on all of the servers. This is reproducible in that I can access the site directly on one server, watch for the file to change and when it does I can refresh the page and receive the error for 1 to 3 seconds.

Some points and theories:

I've suspected this is a permissions issue though that doesn't seem likely. I've exhausted permission configuration options. I've gone so far as to configure the file share with complete public access utilizing the Everyone user account group and the identity account has been given full administrator privileges by being included in the Administrators group. I've adjusted the Authentication ASP.NET impersonation settings among other things.

Suspecting some kind of UNC file share caching I've reviewed the Shadow Copy (VSS) settings on the source share server and I've looked at each web server to ensure there are no previous versions, etc. This also seems unlikely a cause to me.

Running the application in Integrated mode is not an option

This issue occurs on every single one of our 16 web servers and it appears they all (or most) suffer at the same time through the same file change iterations.

To be very clear, this issue only showed itself when I moved the share from our Windows 2003 w/ SQL server to our Windows 2008 w/ SQL server. Configurations are as identical as possible and there is nothing special about the configuration, it's a simple standard file share with common permissions configured.

I've completed the requisite reboots, reconfigurations, service resets, etc.

I've attempted to change the UNC path configuration to a mapped drive configuration but there are application issues that prevent that option currently. I do not know if that would solve this current problem and I'd much prefer not to implement this as a solution.

This issue causes 1000's of errors per day. Hopefully someone has some ideas? I'll be eternally grateful for any help on this!

Justin

RE: Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On UNC Path

[Digy <di...@gmail.com>]

Thanks for sharing your experience.
DIGY

-----Original Message-----
From: Justin Crossman [mailto:justin@deliriousvisions.com] 
Sent: Saturday, December 11, 2010 1:29 AM
To: lucene-net-user@lucene.apache.org
Subject: Re: Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On
UNC Path

I finally found the solution to my issue:

http://technet.microsoft.com/en-us/library/ff686200(WS.10).aspx

Setting DirectoryCacheLifetime [DWORD] to zero did the trick.

On Dec 10, 2010, at 11:04 AM, Justin Crossman wrote:

> I am very new to this list, thank you for indulging me.
> 
> I'm dealing with an issue and have been unable to find any posts or
documentation that already address this specific problem.
> 
> The setup:
> 
> This issue is regarding a heavily accessed web site, written in .NET 3.5
and running on IIS7. The application utilizes a Lucene search index via UNC
path set within the application's Web.config. This UNC path is the same on
each of 16 web servers and they all access the same index in this single
location. The application runs in classic mode (vs integrated) and it uses
an identity and impersonation. The account has sufficient permissions. The
web servers are all currently Windows 2008 servers and the source file share
is also Windows 2008 server running SQL Server 2008 as well. The servers are
always in a Workstation configuration, never a domain and no AD. All servers
have one main user account configured identically with identical permissions
and password.
> 
> The history:
> 
> This setup has been working fine for years under a few configurations.
Recently with web servers running Windows 2003 and IIS6 accessing this
remote UNC file share on a Windows 2003 server running SQL Server 2005. And
most recently with web servers running Windows 2008 and IIS7 accessing this
remote UNC file share on the same Windows 2003 with SQL as just referred to.
The problem occurs only now after having moved this UNC share to this new
Windows 2008 server with SQL.
> 
> The problem:
> 
> The site runs well currently and each web server has access to the index
and successfully utilizes the index as it should. However, periodically
(between 1 and 15 minutes based on activity and a timed service) the Lucene
"segments" file is removed and replaced (it's changed). The current file,
e.g. "segments_xyz" is replaced with "segments_zyx". Not every time this
occurs, but most times, the application seeks out the previous file, not the
current file. This results in a FileNotFoundException and the .NET error
reported is: System.IO.FileNotFoundException: Could not find file
'\X.X.X.X\Index\20101201\guid-x-x-x-x\segments_zyx'. This lasts for 1 to 3
seconds and it occurs on all of the servers. This is reproducible in that I
can access the site directly on one server, watch for the file to change and
when it does I can refresh the page and receive the error for 1 to 3
seconds.
> 
> Some points and theories:
> 
> I've suspected this is a permissions issue though that doesn't seem
likely. I've exhausted permission configuration options. I've gone so far as
to configure the file share with complete public access utilizing the
Everyone user account group and the identity account has been given full
administrator privileges by being included in the Administrators group. I've
adjusted the Authentication ASP.NET impersonation settings among other
things.
> 
> Suspecting some kind of UNC file share caching I've reviewed the Shadow
Copy (VSS) settings on the source share server and I've looked at each web
server to ensure there are no previous versions, etc. This also seems
unlikely a cause to me.
> 
> Running the application in Integrated mode is not an option
> 
> This issue occurs on every single one of our 16 web servers and it appears
they all (or most) suffer at the same time through the same file change
iterations.
> 
> To be very clear, this issue only showed itself when I moved the share
from our Windows 2003 w/ SQL server to our Windows 2008 w/ SQL server.
Configurations are as identical as possible and there is nothing special
about the configuration, it's a simple standard file share with common
permissions configured.
> 
> I've completed the requisite reboots, reconfigurations, service resets,
etc.
> 
> I've attempted to change the UNC path configuration to a mapped drive
configuration but there are application issues that prevent that option
currently. I do not know if that would solve this current problem and I'd
much prefer not to implement this as a solution.
> 
> This issue causes 1000's of errors per day. Hopefully someone has some
ideas? I'll be eternally grateful for any help on this!
> 
> Justin

Re: Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On UNC Path

[Troy Howard <th...@gmail.com>]

You probably should set FileNotFoundCacheLifetime to 0 as well.
FileInfoCacheLifetime is probably not relevant.

We have run into other problems along with this is our proprietary
distributed applications. In our case there is a lot of fast
read/write operations on small files going on which causes us issues,
as well as some long running read/write operations. So we deal with
both very rapid handle allocation and handles with a long-lifetime.
Both run into problems but they are different.

We still haven't completely resolved this for Win2008.

Thanks,
Troy


On Fri, Dec 10, 2010 at 3:28 PM, Justin Crossman
<ju...@deliriousvisions.com> wrote:
> I finally found the solution to my issue:
>
> http://technet.microsoft.com/en-us/library/ff686200(WS.10).aspx
>
> Setting DirectoryCacheLifetime [DWORD] to zero did the trick.
>
> On Dec 10, 2010, at 11:04 AM, Justin Crossman wrote:
>
>> I am very new to this list, thank you for indulging me.
>>
>> I'm dealing with an issue and have been unable to find any posts or documentation that already address this specific problem.
>>
>> The setup:
>>
>> This issue is regarding a heavily accessed web site, written in .NET 3.5 and running on IIS7. The application utilizes a Lucene search index via UNC path set within the application's Web.config. This UNC path is the same on each of 16 web servers and they all access the same index in this single location. The application runs in classic mode (vs integrated) and it uses an identity and impersonation. The account has sufficient permissions. The web servers are all currently Windows 2008 servers and the source file share is also Windows 2008 server running SQL Server 2008 as well. The servers are always in a Workstation configuration, never a domain and no AD. All servers have one main user account configured identically with identical permissions and password.
>>
>> The history:
>>
>> This setup has been working fine for years under a few configurations. Recently with web servers running Windows 2003 and IIS6 accessing this remote UNC file share on a Windows 2003 server running SQL Server 2005. And most recently with web servers running Windows 2008 and IIS7 accessing this remote UNC file share on the same Windows 2003 with SQL as just referred to. The problem occurs only now after having moved this UNC share to this new Windows 2008 server with SQL.
>>
>> The problem:
>>
>> The site runs well currently and each web server has access to the index and successfully utilizes the index as it should. However, periodically (between 1 and 15 minutes based on activity and a timed service) the Lucene "segments" file is removed and replaced (it's changed). The current file, e.g. "segments_xyz" is replaced with "segments_zyx". Not every time this occurs, but most times, the application seeks out the previous file, not the current file. This results in a FileNotFoundException and the .NET error reported is: System.IO.FileNotFoundException: Could not find file '\X.X.X.X\Index\20101201\guid-x-x-x-x\segments_zyx'. This lasts for 1 to 3 seconds and it occurs on all of the servers. This is reproducible in that I can access the site directly on one server, watch for the file to change and when it does I can refresh the page and receive the error for 1 to 3 seconds.
>>
>> Some points and theories:
>>
>> I've suspected this is a permissions issue though that doesn't seem likely. I've exhausted permission configuration options. I've gone so far as to configure the file share with complete public access utilizing the Everyone user account group and the identity account has been given full administrator privileges by being included in the Administrators group. I've adjusted the Authentication ASP.NET impersonation settings among other things.
>>
>> Suspecting some kind of UNC file share caching I've reviewed the Shadow Copy (VSS) settings on the source share server and I've looked at each web server to ensure there are no previous versions, etc. This also seems unlikely a cause to me.
>>
>> Running the application in Integrated mode is not an option
>>
>> This issue occurs on every single one of our 16 web servers and it appears they all (or most) suffer at the same time through the same file change iterations.
>>
>> To be very clear, this issue only showed itself when I moved the share from our Windows 2003 w/ SQL server to our Windows 2008 w/ SQL server. Configurations are as identical as possible and there is nothing special about the configuration, it's a simple standard file share with common permissions configured.
>>
>> I've completed the requisite reboots, reconfigurations, service resets, etc.
>>
>> I've attempted to change the UNC path configuration to a mapped drive configuration but there are application issues that prevent that option currently. I do not know if that would solve this current problem and I'd much prefer not to implement this as a solution.
>>
>> This issue causes 1000's of errors per day. Hopefully someone has some ideas? I'll be eternally grateful for any help on this!
>>
>> Justin
>
>

Re: Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On UNC Path

[Justin Crossman <ju...@deliriousvisions.com>]

I finally found the solution to my issue:

http://technet.microsoft.com/en-us/library/ff686200(WS.10).aspx

Setting DirectoryCacheLifetime [DWORD] to zero did the trick.

On Dec 10, 2010, at 11:04 AM, Justin Crossman wrote:

> I am very new to this list, thank you for indulging me.
> 
> I'm dealing with an issue and have been unable to find any posts or documentation that already address this specific problem.
> 
> The setup:
> 
> This issue is regarding a heavily accessed web site, written in .NET 3.5 and running on IIS7. The application utilizes a Lucene search index via UNC path set within the application's Web.config. This UNC path is the same on each of 16 web servers and they all access the same index in this single location. The application runs in classic mode (vs integrated) and it uses an identity and impersonation. The account has sufficient permissions. The web servers are all currently Windows 2008 servers and the source file share is also Windows 2008 server running SQL Server 2008 as well. The servers are always in a Workstation configuration, never a domain and no AD. All servers have one main user account configured identically with identical permissions and password.
> 
> The history:
> 
> This setup has been working fine for years under a few configurations. Recently with web servers running Windows 2003 and IIS6 accessing this remote UNC file share on a Windows 2003 server running SQL Server 2005. And most recently with web servers running Windows 2008 and IIS7 accessing this remote UNC file share on the same Windows 2003 with SQL as just referred to. The problem occurs only now after having moved this UNC share to this new Windows 2008 server with SQL.
> 
> The problem:
> 
> The site runs well currently and each web server has access to the index and successfully utilizes the index as it should. However, periodically (between 1 and 15 minutes based on activity and a timed service) the Lucene "segments" file is removed and replaced (it's changed). The current file, e.g. "segments_xyz" is replaced with "segments_zyx". Not every time this occurs, but most times, the application seeks out the previous file, not the current file. This results in a FileNotFoundException and the .NET error reported is: System.IO.FileNotFoundException: Could not find file '\X.X.X.X\Index\20101201\guid-x-x-x-x\segments_zyx'. This lasts for 1 to 3 seconds and it occurs on all of the servers. This is reproducible in that I can access the site directly on one server, watch for the file to change and when it does I can refresh the page and receive the error for 1 to 3 seconds.
> 
> Some points and theories:
> 
> I've suspected this is a permissions issue though that doesn't seem likely. I've exhausted permission configuration options. I've gone so far as to configure the file share with complete public access utilizing the Everyone user account group and the identity account has been given full administrator privileges by being included in the Administrators group. I've adjusted the Authentication ASP.NET impersonation settings among other things.
> 
> Suspecting some kind of UNC file share caching I've reviewed the Shadow Copy (VSS) settings on the source share server and I've looked at each web server to ensure there are no previous versions, etc. This also seems unlikely a cause to me.
> 
> Running the application in Integrated mode is not an option
> 
> This issue occurs on every single one of our 16 web servers and it appears they all (or most) suffer at the same time through the same file change iterations.
> 
> To be very clear, this issue only showed itself when I moved the share from our Windows 2003 w/ SQL server to our Windows 2008 w/ SQL server. Configurations are as identical as possible and there is nothing special about the configuration, it's a simple standard file share with common permissions configured.
> 
> I've completed the requisite reboots, reconfigurations, service resets, etc.
> 
> I've attempted to change the UNC path configuration to a mapped drive configuration but there are application issues that prevent that option currently. I do not know if that would solve this current problem and I'd much prefer not to implement this as a solution.
> 
> This issue causes 1000's of errors per day. Hopefully someone has some ideas? I'll be eternally grateful for any help on this!
> 
> Justin

Re: Lucene | IIS7 .NET Web Site Delays Detection Of File Changes On UNC Path

[Troy Howard <th...@gmail.com>]

I can't comment on Lucene.Net specific issues but I can say that using
Windows 2008 as a file server has been problematic for our company.

I would focus your inquiry into configuring Windows 2008. We had
problems with unreleased file handles and delayed writes in a
high-traffic file server. This could cause the kind of behaviour
you're seeing. Maybe move to a more solid OS (eg, non-windows) for
your file share.

Thanks,
Troy

On Fri, Dec 10, 2010 at 10:04 AM, Justin Crossman
<ju...@deliriousvisions.com> wrote:
> I am very new to this list, thank you for indulging me.
>
> I'm dealing with an issue and have been unable to find any posts or documentation that already address this specific problem.
>
> The setup:
>
> This issue is regarding a heavily accessed web site, written in .NET 3.5 and running on IIS7. The application utilizes a Lucene search index via UNC path set within the application's Web.config. This UNC path is the same on each of 16 web servers and they all access the same index in this single location. The application runs in classic mode (vs integrated) and it uses an identity and impersonation. The account has sufficient permissions. The web servers are all currently Windows 2008 servers and the source file share is also Windows 2008 server running SQL Server 2008 as well. The servers are always in a Workstation configuration, never a domain and no AD. All servers have one main user account configured identically with identical permissions and password.
>
> The history:
>
> This setup has been working fine for years under a few configurations. Recently with web servers running Windows 2003 and IIS6 accessing this remote UNC file share on a Windows 2003 server running SQL Server 2005. And most recently with web servers running Windows 2008 and IIS7 accessing this remote UNC file share on the same Windows 2003 with SQL as just referred to. The problem occurs only now after having moved this UNC share to this new Windows 2008 server with SQL.
>
> The problem:
>
> The site runs well currently and each web server has access to the index and successfully utilizes the index as it should. However, periodically (between 1 and 15 minutes based on activity and a timed service) the Lucene "segments" file is removed and replaced (it's changed). The current file, e.g. "segments_xyz" is replaced with "segments_zyx". Not every time this occurs, but most times, the application seeks out the previous file, not the current file. This results in a FileNotFoundException and the .NET error reported is: System.IO.FileNotFoundException: Could not find file '\X.X.X.X\Index\20101201\guid-x-x-x-x\segments_zyx'. This lasts for 1 to 3 seconds and it occurs on all of the servers. This is reproducible in that I can access the site directly on one server, watch for the file to change and when it does I can refresh the page and receive the error for 1 to 3 seconds.
>
> Some points and theories:
>
> I've suspected this is a permissions issue though that doesn't seem likely. I've exhausted permission configuration options. I've gone so far as to configure the file share with complete public access utilizing the Everyone user account group and the identity account has been given full administrator privileges by being included in the Administrators group. I've adjusted the Authentication ASP.NET impersonation settings among other things.
>
> Suspecting some kind of UNC file share caching I've reviewed the Shadow Copy (VSS) settings on the source share server and I've looked at each web server to ensure there are no previous versions, etc. This also seems unlikely a cause to me.
>
> Running the application in Integrated mode is not an option
>
> This issue occurs on every single one of our 16 web servers and it appears they all (or most) suffer at the same time through the same file change iterations.
>
> To be very clear, this issue only showed itself when I moved the share from our Windows 2003 w/ SQL server to our Windows 2008 w/ SQL server. Configurations are as identical as possible and there is nothing special about the configuration, it's a simple standard file share with common permissions configured.
>
> I've completed the requisite reboots, reconfigurations, service resets, etc.
>
> I've attempted to change the UNC path configuration to a mapped drive configuration but there are application issues that prevent that option currently. I do not know if that would solve this current problem and I'd much prefer not to implement this as a solution.
>
> This issue causes 1000's of errors per day. Hopefully someone has some ideas? I'll be eternally grateful for any help on this!
>
> Justin