You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@commons.apache.org by "Juha-Matti Toppinen (JIRA)" <ji...@apache.org> on 2006/10/31 15:56:28 UTC
[jira] Created: (VFS-98) VFS causes deadlocks or is not thread-safe
VFS causes deadlocks or is not thread-safe
------------------------------------------
Key: VFS-98
URL: http://issues.apache.org/jira/browse/VFS-98
Project: Commons VFS
Issue Type: Bug
Affects Versions: Nightly Builds
Environment: jdk1.5.0_07 / Linux
Reporter: Juha-Matti Toppinen
Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
I have tried, without success different workarounds based on some mailing list threads:
- Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
- Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
The version I have problems with is a nightly build: commons-vfs-20060831
The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498957 ]
Mario Ivankovits commented on VFS-98:
-------------------------------------
Yes, we fixed it.
If you experience some deadlocks try generating a "full thread dump". Its output should help greatly :-)
Ciao,
Mario
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Chris Beams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12498825 ]
Chris Beams commented on VFS-98:
--------------------------------
I assume that because this issue was resolved before the release date of Commons VFS 1.0, that this fix is included in 1.0. However, the Fix Version is not explicit about this, so I'm writing to make sure my assumption is correct. I'm running into thread-related issues as well, and though I haven't pinpointed them, they sound similar to those Juha-Matti was experiencing.
Thanks!
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Juha-Matti Toppinen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=comments#action_12448056 ]
Juha-Matti Toppinen commented on VFS-98:
----------------------------------------
Your fix indeed does seem to fix the problem. This issue, as well as the solution was also quite educational experience for me. Thank you very much!
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Chris Beams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506958 ]
Chris Beams commented on VFS-98:
--------------------------------
Mario,
Thanks so much for this (simple!) fix... We have been testing it very heavily these past two days, and have not encountered a single deadlock.
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Chris Beams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12501017 ]
Chris Beams commented on VFS-98:
--------------------------------
Mario,
Any other thoughts on this topic? As Larry mentioned, we do have a workaround in place, but it creates a severe performance penalty. We've synchronized all access to the VFS FileSystemManager, so all actions are serialized. We're in a large-volume multithreaded environment, so this quick fix isn't going to hold up for long.
VFS is so valuable to us that, if neccessary, we may go down the road of implementing a proper fix ourselves. Not knowing the codebase, however, this could take quite a bit of guesswork and time. Any help you can provide would be most appreciated.
Thanks - Chris
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=comments#action_12445918 ]
Mario Ivankovits commented on VFS-98:
-------------------------------------
I dont know why, but everytime when I'll go on vacation there is something happening around vfs ;-)
First, please stick with the SoftReferenceFilesCache, its your only option in reality.
Then, please try one of the very latest builds, as far as I remember I fixed something in this area, but can't remember when it was, though, I am pretty sure it will help here.
When your application hangs anyway, please create a "full thread dump" [1] and post it here. This should allow us to debug your problem.
As a workaround you can create a FileSystemManager per thread by
1) create the StandardFileSystem manager manually (do not user VFS.getManager())
2) store the reference in an java.lang.ThreadLocal
Hope this helps for now.
[1] http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=comments#action_12447937 ]
Mario Ivankovits commented on VFS-98:
-------------------------------------
Hi!
I committed a fix for this .... hopefully.
Could you please try it again. In case it hangs again please attach the stacktrace again, but not the jstack one, it doesn't contain enough informations.
Thanks!
Ciao,
Mario
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Juha-Matti Toppinen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=comments#action_12446226 ]
Juha-Matti Toppinen commented on VFS-98:
----------------------------------------
Thank you for the quick response.
Creating a per-thread FileSystemManager is not very comfortable, because I have many, many classes that are shared between threads, and they all have a private main FileObject it uses to store information etc. They also are maintained using Multiton -like pattern, so that only one shared instance is created per FileObject, using FileObject.toString() as object pool key. Redesigning all these classes to use per-thread FileSystemManagers as well as per-thread FileObjects would be quite a lot of work.
I checked out and built VFS from SVN (nightly builds directory was empty) to get the latest version, and still had the same problem.
I attach both a thread dump (thanks for the helpful link) as well as a report generated by jstack.
I tried both with the default VFS.getManager() and creating a custom DefaultFileSystemManager that uses SynchronizedFileObject decorator. Only difference was that with SynchronizedFileObject decorator the deadlock situation occurred more quickly.
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506089 ]
Mario Ivankovits commented on VFS-98:
-------------------------------------
First: Sorry for my long delay!
I think I managed to fix this locking issue, could you please give VFS HEAD a try.
Thanks!
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Resolved: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=all ]
Mario Ivankovits resolved VFS-98.
---------------------------------
Resolution: Fixed
You are welcome :-)
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Chris Beams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12505882 ]
Chris Beams commented on VFS-98:
--------------------------------
Mario, all,
We are running into these threading issues (deadlocks) right and left, and trying to work around them by synchronizing on the default shared VFS FileSystemManager instance pretty much everywhere. Even this is not working, however, and we're at a crossroads with what seems to be three options: 1) Get help from the VFS team 2) Attempt to fix the issues ourselves 3) ditch VFS and use lower-level libraries directly.
We'd like to avoid (3) if at all possible; VFS rocks in terms of functionality, and saves us a great deal of coding. We're willing to look into option (2), but we'd like to have some design-level conversations with you first. If there are just 'holes' in VFS's thread-safety that can be patched up, we can work with that; if there are fundamental design problems that preclude thread-safety, it's probably going to be beyond reasonable scope for us to work on.
We can also attempt to work up some simple examples that have high likelihood of exhibiting the deadlocks we're seeing.
At any rate, we'd really appreciate some feedback on this issue so we can quickly choose one of the three options above.
Thanks again!
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Updated: (VFS-98) VFS causes deadlocks or is not thread-safe
Posted by "Larry Johnson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Larry Johnson updated VFS-98:
-----------------------------
Attachment: vfs_deadlock.txt
Hi Mario,
Attached is an example of the deadlock Chris mentioned. To give you some background info all threads/FileObject operations in this thread dump used the same VFS.getManager() instance. We've been able to get around this for the time being by synchronizing all manager/fileobject operations in a separate singleton class.
Thanks - Larry
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Mario Ivankovits (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/VFS-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12506967 ]
Mario Ivankovits commented on VFS-98:
-------------------------------------
Hehe - sometimes something looks complicated - its great if there is a simple fix for such a case :-)
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: https://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assignee: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt, vfs_deadlock.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org
[jira] Commented: (VFS-98) VFS causes deadlocks or is not
thread-safe
Posted by "Juha-Matti Toppinen (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/VFS-98?page=comments#action_12447180 ]
Juha-Matti Toppinen commented on VFS-98:
----------------------------------------
Hi again.
I had some progress on this issue. The problem seems to occur only when a single FileObject is accessed from multiple threads. It works happily when using a single FileSystem, but accessing different files in it simultaneously.
For me the possible one-time errors caused by the lack of proper synchronization do not matter so much. But a deadlock on the FileSystem is really bad, causing may server to completely stop responding. So I played around trying to fix the problem by just commenting out syncronization stuff based on threaddump information. Not a the most professional method of working around a bug, but I desperately needed a workaround ;)
Eventually I got it working, and I'll just list the not-so-well-thought modifications I made, so you can maybe take a look at them and possibly use them as a basis for a proper fix...
AbstractFileProvider:
-------------------------------------------------------------------
- replaced private final Map fileSystems = new TreeMap(); with: private final Map fileSystems = java.util.Collections.synchronizedMap(new TreeMap()); in order to keep some thread-safety.
- commented out synchronization stuff from close(), addFileSystem(), findFileSystem() and freeUnusedResources()
These did not help yet, so I also made these modifications:
AbstractFileObject:
-------------------------------------------------------------------
- commented out synchronization stuff from detach()
AbstractFileSystem:
-------------------------------------------------------------------
- removed synchronized keywords from resolveFile(final FileName name) and resolveFile(final FileName name, final boolean useCache)
And no deadlocks anymore. Hope this helps!
> VFS causes deadlocks or is not thread-safe
> ------------------------------------------
>
> Key: VFS-98
> URL: http://issues.apache.org/jira/browse/VFS-98
> Project: Commons VFS
> Issue Type: Bug
> Affects Versions: Nightly Builds
> Environment: jdk1.5.0_07 / Linux
> Reporter: Juha-Matti Toppinen
> Assigned To: Mario Ivankovits
> Attachments: vfs.dead.jstack.txt, vfs.dead.threaddump.synchronizedfileobject.txt, vfs.dead.threaddump.txt
>
>
> Newer versions of VFS seems to be unusable in multithreading environments, causing a deadlock of some kind. This causes my Java web server application to completely hang when there is many concurrent connections. My application uses a single FileSystemManager instance, and makes quite heavy use of VFS; opening many files from many threads simultaneously.
> I have tried, without success different workarounds based on some mailing list threads:
> - Using both NullReferenceFilesCache and the default SoftReferenceFilesCache. (SoftReferenceFilesCache seemed to sometimes cause some additional thread-related exceptions under heavy load, but propably unrelated to this issue)
> - Using the new SynchorizedFileObjectDecorator also did not help. On the contrary, it seemed to trigger the deadlock more easily.
> The version I have problems with is a nightly build: commons-vfs-20060831
> The older version commons-vfs-20060115 works beautifully, but I would like to use newer version because I want to be able to use FileObject.refresh() to reset the internal state of files (specially, to get a fresh list of children).
> I hope the threading issues are going to be resolved in near future, and I am willing to help in any way. I do not have very deep experience about multithreading applications, so I wasn't able to get more close to the roots of the issue. Feel free to ask for more information.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-dev-help@jakarta.apache.org