You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Juergen Mayrbaeurl (JIRA)" <ji...@apache.org> on 2007/06/29 11:40:33 UTC
[jira] Created: (SM-990) FilePoller with Archiving
FilePoller with Archiving
-------------------------
Key: SM-990
URL: https://issues.apache.org/activemq/browse/SM-990
Project: ServiceMix
Issue Type: Improvement
Components: servicemix-components, servicemix-file
Affects Versions: 3.1
Reporter: Juergen Mayrbaeurl
Priority: Minor
The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Gert Vanthienen <ge...@skynet.be>.
Guillaume Nodet wrote:
> On 8/9/07, Gert Vanthienen <ge...@skynet.be> wrote:
>
>> Guillaume Nodet wrote:
>>
>>> You mean a file based implementation of servicemix-audit ?
>>> See http://incubator.apache.org/servicemix/dist/servicemix-3.1-incubating/site/core/servicemix-audit/apidocs/org/apache/servicemix/jbi/audit/AuditorMBean.html
>>>
>>>
>> Yes, my first comment was exactly that: using an auditor for archiving
>> the files. This isn't the best solution for this particular issue, but
>> it still might be a worthwhile addition to ServiceMix. We still need to
>> find a good way to avoid having to load entire messages into memory
>> however. Anybody has any good suggestions for solving that?
>>
>
> I think a TeeInputStream should work perfectly. I guess this would be
> the best option for very large files (it would avoid reading the whole
> file twice).
>
>
>>> I think we should only backport bug fixes, do you ?
>>> But I'm open to discuss that, especially given the long release cycle
>>> we have (we really need to do something about that, btw).
>>>
>>>
>> That's exactly why I was asking. I think these branches are great for
>> doing bug fixes on an existing release, but I agree we should try to
>> avoid it for new features. What is holding us back to shorten the
>> release cycle (or even time-boxing it)? I'm trying to build a tool to
>> regenerate the legal files, so we can already avoid the pain of checking
>> those 'manually' every time. Is there anything else?
>>
>>
>
> Nothing, really. I'd like the new endpoints for http and jms be finished, but
> nothing else. However it may be worth waiting for the board meeting (one or two
> weeks) so that we can remove the incubation disclaimers and have a release that
> we can put on maven public repositories. Wdyt?
It probably is worth waiting for then. I think we should even consider
releasing 3.2 (with the new HTTP/JMS endpoints, servicemix-camel, ...)
instead of another 3.1.x bugfix release.
There is another issue I would like to see resolved in our next release,
which is the one mentioned in
http://www.nabble.com/ActiveMQ-Causing-OutOfMemoryError-After-Service-Deployment-tf4133779s12049.html.
Does anyone know what the corresponding ActiveMQ issue is and when it
will become available in a release?
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Guillaume Nodet <gn...@gmail.com>.
On 8/9/07, Gert Vanthienen <ge...@skynet.be> wrote:
>
>
> Guillaume Nodet wrote:
> > You mean a file based implementation of servicemix-audit ?
> > See http://incubator.apache.org/servicemix/dist/servicemix-3.1-incubating/site/core/servicemix-audit/apidocs/org/apache/servicemix/jbi/audit/AuditorMBean.html
> >
> Yes, my first comment was exactly that: using an auditor for archiving
> the files. This isn't the best solution for this particular issue, but
> it still might be a worthwhile addition to ServiceMix. We still need to
> find a good way to avoid having to load entire messages into memory
> however. Anybody has any good suggestions for solving that?
I think a TeeInputStream should work perfectly. I guess this would be
the best option for very large files (it would avoid reading the whole
file twice).
> > I think we should only backport bug fixes, do you ?
> > But I'm open to discuss that, especially given the long release cycle
> > we have (we really need to do something about that, btw).
> >
> That's exactly why I was asking. I think these branches are great for
> doing bug fixes on an existing release, but I agree we should try to
> avoid it for new features. What is holding us back to shorten the
> release cycle (or even time-boxing it)? I'm trying to build a tool to
> regenerate the legal files, so we can already avoid the pain of checking
> those 'manually' every time. Is there anything else?
>
Nothing, really. I'd like the new endpoints for http and jms be finished, but
nothing else. However it may be worth waiting for the board meeting (one or two
weeks) so that we can remove the incubation disclaimers and have a release that
we can put on maven public repositories. Wdyt?
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Gert Vanthienen <ge...@skynet.be>.
Guillaume Nodet wrote:
> You mean a file based implementation of servicemix-audit ?
> See http://incubator.apache.org/servicemix/dist/servicemix-3.1-incubating/site/core/servicemix-audit/apidocs/org/apache/servicemix/jbi/audit/AuditorMBean.html
>
Yes, my first comment was exactly that: using an auditor for archiving
the files. This isn't the best solution for this particular issue, but
it still might be a worthwhile addition to ServiceMix. We still need to
find a good way to avoid having to load entire messages into memory
however. Anybody has any good suggestions for solving that?
> I think we should only backport bug fixes, do you ?
> But I'm open to discuss that, especially given the long release cycle
> we have (we really need to do something about that, btw).
>
That's exactly why I was asking. I think these branches are great for
doing bug fixes on an existing release, but I agree we should try to
avoid it for new features. What is holding us back to shorten the
release cycle (or even time-boxing it)? I'm trying to build a tool to
regenerate the legal files, so we can already avoid the pain of checking
those 'manually' every time. Is there anything else?
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Guillaume Nodet <gn...@gmail.com>.
On 8/8/07, Gert Vanthienen <ge...@skynet.be> wrote:
> Guillaume,
>
> I was thinking about creating a generic archiving service, which would
> be able to keep an archive of every message that has been processed by
> ServiceMix for auditing purposes (and e.g. to be able to re-send it when
> necessary). In this use case, it would be very useful to have a single
> point of configuration instead of having to configure the archive flag
> over and over again on all the consumer endpoints in an application.
You mean a file based implementation of servicemix-audit ?
See http://incubator.apache.org/servicemix/dist/servicemix-3.1-incubating/site/core/servicemix-audit/apidocs/org/apache/servicemix/jbi/audit/AuditorMBean.html
>
> But this is probably an entirely different requirement than the one
> expressed by this user. I'll add an 'archive' attribute to the poller
> endpoint to refer to the archiving directory for this issue. Is this
> something to backport to 3.1.2-SNAPSHOT as well?
I think we should only backport bug fixes, do you ?
But I'm open to discuss that, especially given the long release cycle
we have (we really need to do something about that, btw).
>
> Gert
>
>
> Guillaume Nodet wrote:
> > It should be quite easy to write a TeeInputStream that would write to
> > an OutputStream each time a charachter is read from the input stream.
> > However, the content of the NormalizedMessage is a Source and may not
> > be a stream.
> >
> > However, I think it would be easier to put this feature inside the BC
> > rather than in an intermediate (a ExchangeListener on the container?).
> > Well, at least, it would be easier to configure as a simple flag on
> > the endpoint could work. If we use a single listener, we will have to
> > configure on the main servicemix.xml configuration file or register it
> > dynamically (from were?) ...
> >
> > On 8/8/07, Gert Vanthienen (JIRA) <ji...@apache.org> wrote:
> >
> >> [ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39880 ]
> >>
> >>
> >> Gert Vanthienen commented on SM-990:
> >> ------------------------------------
> >>
> >>
> >> I was thinking about some kind of WireTap (implemented as an Auditor to easily apply it to all message), but I suppose you're worrying about the fact that we need to ensure re-readability of the stream for this. Would it be possible to 'decorate' the NormalizedMessage somehow, so it can intercept when bytes are read from the inputstream by the provider and do the archiving on-the-fly?
> >>
> >>
> >>> FilePoller with Archiving
> >>> -------------------------
> >>>
> >>> Key: SM-990
> >>> URL: https://issues.apache.org/activemq/browse/SM-990
> >>> Project: ServiceMix
> >>> Issue Type: Improvement
> >>> Components: servicemix-components, servicemix-file
> >>> Affects Versions: 3.1
> >>> Reporter: Juergen Mayrbaeurl
> >>> Priority: Minor
> >>>
> >>> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
> >>>
> >> --
> >> This message is automatically generated by JIRA.
> >> -
> >> You can reply to this email to add a comment to the issue online.
> >>
> >>
> >>
> >
> >
> >
>
--
Cheers,
Guillaume Nodet
------------------------
Blog: http://gnodet.blogspot.com/
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Gert Vanthienen <ge...@skynet.be>.
Guillaume,
I was thinking about creating a generic archiving service, which would
be able to keep an archive of every message that has been processed by
ServiceMix for auditing purposes (and e.g. to be able to re-send it when
necessary). In this use case, it would be very useful to have a single
point of configuration instead of having to configure the archive flag
over and over again on all the consumer endpoints in an application.
But this is probably an entirely different requirement than the one
expressed by this user. I'll add an 'archive' attribute to the poller
endpoint to refer to the archiving directory for this issue. Is this
something to backport to 3.1.2-SNAPSHOT as well?
Gert
Guillaume Nodet wrote:
> It should be quite easy to write a TeeInputStream that would write to
> an OutputStream each time a charachter is read from the input stream.
> However, the content of the NormalizedMessage is a Source and may not
> be a stream.
>
> However, I think it would be easier to put this feature inside the BC
> rather than in an intermediate (a ExchangeListener on the container?).
> Well, at least, it would be easier to configure as a simple flag on
> the endpoint could work. If we use a single listener, we will have to
> configure on the main servicemix.xml configuration file or register it
> dynamically (from were?) ...
>
> On 8/8/07, Gert Vanthienen (JIRA) <ji...@apache.org> wrote:
>
>> [ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39880 ]
>>
>>
>> Gert Vanthienen commented on SM-990:
>> ------------------------------------
>>
>>
>> I was thinking about some kind of WireTap (implemented as an Auditor to easily apply it to all message), but I suppose you're worrying about the fact that we need to ensure re-readability of the stream for this. Would it be possible to 'decorate' the NormalizedMessage somehow, so it can intercept when bytes are read from the inputstream by the provider and do the archiving on-the-fly?
>>
>>
>>> FilePoller with Archiving
>>> -------------------------
>>>
>>> Key: SM-990
>>> URL: https://issues.apache.org/activemq/browse/SM-990
>>> Project: ServiceMix
>>> Issue Type: Improvement
>>> Components: servicemix-components, servicemix-file
>>> Affects Versions: 3.1
>>> Reporter: Juergen Mayrbaeurl
>>> Priority: Minor
>>>
>>> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
>>>
>> --
>> This message is automatically generated by JIRA.
>> -
>> You can reply to this email to add a comment to the issue online.
>>
>>
>>
>
>
>
Re: [jira] Commented: (SM-990) FilePoller with Archiving
Posted by Guillaume Nodet <gn...@gmail.com>.
It should be quite easy to write a TeeInputStream that would write to
an OutputStream each time a charachter is read from the input stream.
However, the content of the NormalizedMessage is a Source and may not
be a stream.
However, I think it would be easier to put this feature inside the BC
rather than in an intermediate (a ExchangeListener on the container?).
Well, at least, it would be easier to configure as a simple flag on
the endpoint could work. If we use a single listener, we will have to
configure on the main servicemix.xml configuration file or register it
dynamically (from were?) ...
On 8/8/07, Gert Vanthienen (JIRA) <ji...@apache.org> wrote:
>
> [ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39880 ]
>
>
> Gert Vanthienen commented on SM-990:
> ------------------------------------
>
>
> I was thinking about some kind of WireTap (implemented as an Auditor to easily apply it to all message), but I suppose you're worrying about the fact that we need to ensure re-readability of the stream for this. Would it be possible to 'decorate' the NormalizedMessage somehow, so it can intercept when bytes are read from the inputstream by the provider and do the archiving on-the-fly?
>
> > FilePoller with Archiving
> > -------------------------
> >
> > Key: SM-990
> > URL: https://issues.apache.org/activemq/browse/SM-990
> > Project: ServiceMix
> > Issue Type: Improvement
> > Components: servicemix-components, servicemix-file
> > Affects Versions: 3.1
> > Reporter: Juergen Mayrbaeurl
> > Priority: Minor
> >
> > The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
--
Cheers,
Guillaume Nodet
------------------------
Principal Engineer, IONA
Blog: http://gnodet.blogspot.com/
[jira] Commented: (SM-990) FilePoller with Archiving
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39877 ]
Guillaume Nodet commented on SM-990:
------------------------------------
You mean a kind of EIP wire tap ?
We need to find a way to deal with big files too and avoiding loading them in memory if possible.
> FilePoller with Archiving
> -------------------------
>
> Key: SM-990
> URL: https://issues.apache.org/activemq/browse/SM-990
> Project: ServiceMix
> Issue Type: Improvement
> Components: servicemix-components, servicemix-file
> Affects Versions: 3.1
> Reporter: Juergen Mayrbaeurl
> Priority: Minor
>
> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-990) FilePoller with Archiving
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39651 ]
Gert Vanthienen commented on SM-990:
------------------------------------
Perhaps we can create a more generic solution for this: with SM-751 we have a correlation id that allows us to do flow tracing, so it should be possible to build an auditor that is triggered whenever a new message arrives (at any kind of endpoint: FTP, File, HTTP, ...) and sends a copy of the message to an archiving endpoint.
> FilePoller with Archiving
> -------------------------
>
> Key: SM-990
> URL: https://issues.apache.org/activemq/browse/SM-990
> Project: ServiceMix
> Issue Type: Improvement
> Components: servicemix-components, servicemix-file
> Affects Versions: 3.1
> Reporter: Juergen Mayrbaeurl
> Priority: Minor
>
> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-990) FilePoller with Archiving
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_39880 ]
Gert Vanthienen commented on SM-990:
------------------------------------
I was thinking about some kind of WireTap (implemented as an Auditor to easily apply it to all message), but I suppose you're worrying about the fact that we need to ensure re-readability of the stream for this. Would it be possible to 'decorate' the NormalizedMessage somehow, so it can intercept when bytes are read from the inputstream by the provider and do the archiving on-the-fly?
> FilePoller with Archiving
> -------------------------
>
> Key: SM-990
> URL: https://issues.apache.org/activemq/browse/SM-990
> Project: ServiceMix
> Issue Type: Improvement
> Components: servicemix-components, servicemix-file
> Affects Versions: 3.1
> Reporter: Juergen Mayrbaeurl
> Priority: Minor
>
> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SM-990) FilePoller with Archiving
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gert Vanthienen resolved SM-990.
--------------------------------
Resolution: Fixed
Fix Version/s: 3.2
Added {{archive}} attribute to specify the archive directory
http://svn.apache.org/viewvc?view=rev&revision=564081
> FilePoller with Archiving
> -------------------------
>
> Key: SM-990
> URL: https://issues.apache.org/activemq/browse/SM-990
> Project: ServiceMix
> Issue Type: Improvement
> Components: servicemix-components, servicemix-file
> Affects Versions: 3.1
> Reporter: Juergen Mayrbaeurl
> Priority: Minor
> Fix For: 3.2
>
>
> The various FilePoller implementations should archive the files before deleting them (e.g. by copying them to another directory)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.