You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@felix.apache.org by "Peter Lawrey (JIRA)" <ji...@apache.org> on 2010/06/22 17:52:54 UTC
[jira] Created: (FELIX-2436) Reduce the number files created by the
fileinstall Scanner.
Reduce the number files created by the fileinstall Scanner.
-----------------------------------------------------------
Key: FELIX-2436
URL: https://issues.apache.org/jira/browse/FELIX-2436
Project: Felix
Issue Type: Improvement
Components: File Install
Affects Versions: fileinstall-2.0.8
Reporter: Peter Lawrey
The fileinstaller is called regularly and there can be two or more instances in a server.
However its object allocation doesn't considered that 99.999+% of the time none of the files change.
(polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (FELIX-2436) Reduce the number files created by the
fileinstall Scanner.
Posted by "Peter Lawrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Lawrey updated FELIX-2436:
--------------------------------
Attachment: reduce_object_construction.patch
patch of changes to reduce object allocation.
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Peter Lawrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881233#action_12881233 ]
Peter Lawrey commented on FELIX-2436:
-------------------------------------
Unfortunately File creates lots of objects and there is no alternative. It is the subject of this bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6963022
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Peter Lawrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881690#action_12881690 ]
Peter Lawrey commented on FELIX-2436:
-------------------------------------
jnotify does sound like a better solution. There would have to be some fallback impliementation for platforms not supported.
In the meantime I have found that just polling the modification date of the directory is far more light weight and gives me what I need (I don't need sub directories) If the directory has changed I do a full check in the same manner it did before.
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Peter Lawrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881701#action_12881701 ]
Peter Lawrey commented on FELIX-2436:
-------------------------------------
Using the last modified date of the directory cut the objects created to 1/65th of what it was. ( from 40% of objects to 0.6%)
For my usecase, most changes involve files being removed or add and the rest of the time I can do a "touch ."
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924934#action_12924934 ]
Guillaume Nodet commented on FELIX-2436:
----------------------------------------
See https://sourceforge.net/projects/jnotify/forums/forum/515553/topic/3913756
If there's a sufficient interest, it should be possible to use JNotifiy, as the owner is willing to dual license if needed.
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Holger Hoffstätte (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881239#action_12881239 ]
Holger Hoffstätte commented on FELIX-2436:
------------------------------------------
A much better solution would be to turn jnotify (http://jnotify.sourceforge.net/) into a bundle and let the OS do the heavy lifting instead.
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924882#action_12924882 ]
Guillaume Nodet commented on FELIX-2436:
----------------------------------------
I think checking the length of the file is important, especially when you copy a big file and that the write isn't atomic enough. For example when copying a big war, if the copy is still running while the scan is done and that the copy takes more time than the scan interval, there are good chances the check will return true, while the file isn't fully copied.
So i'm not 100% confident with applying this patch...
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (FELIX-2436) Reduce the number files created
by the fileinstall Scanner.
Posted by Felix Meschberger <fm...@gmail.com>.
Cool ! Thanks.
Regards
Felix
On 26.10.2010 11:29, Guillaume Nodet wrote:
> See https://sourceforge.net/projects/jnotify/forums/forum/515553/topic/3913756
>
> On Tue, Oct 26, 2010 at 10:35, Guillaume Nodet <gn...@gmail.com> wrote:
>> Yeah, I was wondering about that, or eventually rewriting something
>> with hawtjni (which is awesome when writing jni code).
>>
>> On Tue, Oct 26, 2010 at 10:23, Felix Meschberger <fm...@gmail.com> wrote:
>>> Hi,
>>>
>>> On 26.10.2010 10:05, Guillaume Nodet (JIRA) wrote:
>>>> JNotify would be a nice idea if it wasn't LGPL which is incompatible with the ASF policy.
>>>
>>> What I was always looking for ...
>>>
>>> Maybe we can get this guy to change the license or at least dual-license
>>> it ?
>>>
>>> Regards
>>> Felix
>>>
>>
>>
>>
>> --
>> Cheers,
>> Guillaume Nodet
>> ------------------------
>> Blog: http://gnodet.blogspot.com/
>> ------------------------
>> Open Source SOA
>> http://fusesource.com
>>
>
>
>
Re: [jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by Guillaume Nodet <gn...@gmail.com>.
See https://sourceforge.net/projects/jnotify/forums/forum/515553/topic/3913756
On Tue, Oct 26, 2010 at 10:35, Guillaume Nodet <gn...@gmail.com> wrote:
> Yeah, I was wondering about that, or eventually rewriting something
> with hawtjni (which is awesome when writing jni code).
>
> On Tue, Oct 26, 2010 at 10:23, Felix Meschberger <fm...@gmail.com> wrote:
>> Hi,
>>
>> On 26.10.2010 10:05, Guillaume Nodet (JIRA) wrote:
>>> JNotify would be a nice idea if it wasn't LGPL which is incompatible with the ASF policy.
>>
>> What I was always looking for ...
>>
>> Maybe we can get this guy to change the license or at least dual-license
>> it ?
>>
>> Regards
>> Felix
>>
>
>
>
> --
> Cheers,
> Guillaume Nodet
> ------------------------
> Blog: http://gnodet.blogspot.com/
> ------------------------
> Open Source SOA
> http://fusesource.com
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: [jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by Guillaume Nodet <gn...@gmail.com>.
Yeah, I was wondering about that, or eventually rewriting something
with hawtjni (which is awesome when writing jni code).
On Tue, Oct 26, 2010 at 10:23, Felix Meschberger <fm...@gmail.com> wrote:
> Hi,
>
> On 26.10.2010 10:05, Guillaume Nodet (JIRA) wrote:
>> JNotify would be a nice idea if it wasn't LGPL which is incompatible with the ASF policy.
>
> What I was always looking for ...
>
> Maybe we can get this guy to change the license or at least dual-license
> it ?
>
> Regards
> Felix
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
------------------------
Open Source SOA
http://fusesource.com
Re: [jira] Commented: (FELIX-2436) Reduce the number files created
by the fileinstall Scanner.
Posted by Felix Meschberger <fm...@gmail.com>.
Hi,
On 26.10.2010 10:05, Guillaume Nodet (JIRA) wrote:
> JNotify would be a nice idea if it wasn't LGPL which is incompatible with the ASF policy.
What I was always looking for ...
Maybe we can get this guy to change the license or at least dual-license
it ?
Regards
Felix
[jira] Commented: (FELIX-2436) Reduce the number files created by
the fileinstall Scanner.
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12924885#action_12924885 ]
Guillaume Nodet commented on FELIX-2436:
----------------------------------------
JNotify would be a nice idea if it wasn't LGPL which is incompatible with the ASF policy.
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (FELIX-2436) Reduce the number files
created by the fileinstall Scanner.
Posted by "Peter Lawrey (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/FELIX-2436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12881233#action_12881233 ]
Peter Lawrey edited comment on FELIX-2436 at 6/22/10 11:59 AM:
---------------------------------------------------------------
Unfortunately java.io.File creates lots of objects and there is no alternative. It is the subject of this bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6963022
was (Author: peter_edgeci):
Unfortunately File creates lots of objects and there is no alternative. It is the subject of this bug http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6963022
> Reduce the number files created by the fileinstall Scanner.
> -----------------------------------------------------------
>
> Key: FELIX-2436
> URL: https://issues.apache.org/jira/browse/FELIX-2436
> Project: Felix
> Issue Type: Improvement
> Components: File Install
> Affects Versions: fileinstall-2.0.8
> Reporter: Peter Lawrey
> Attachments: reduce_object_construction.patch
>
>
> The fileinstaller is called regularly and there can be two or more instances in a server.
> However its object allocation doesn't considered that 99.999+% of the time none of the files change.
> (polling one per second and changing files once per day is means the file installer will see a change 0.001% of the time)
> In our application, polling every second results in the Scanner creating more objects than the application itself. We can turn down the polling rate but even at once every 10s it creates more objects than any other component!
> For this reason I have changed the Scanner to first check whether any files have changed attempting to compute what has changed.
> See atached for a patch of the changes. This reduces the number of obejcts created by atleast a factor of 3.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.