You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Ron Gavlin (JIRA)" <ji...@apache.org> on 2008/03/27 11:29:32 UTC
[jira] Created: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Handle clustering of outbound endpoints using "persisted sequence numbering"
----------------------------------------------------------------------------
Key: SM-1292
URL: https://issues.apache.org/activemq/browse/SM-1292
Project: ServiceMix
Issue Type: New Feature
Components: servicemix-file
Affects Versions: 3.2.1
Reporter: Ron Gavlin
The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=41841#action_41841 ]
Ron Gavlin commented on SM-1292:
--------------------------------
Are you suggesting I Spring dependency-inject some type of DbHelper that is used to populate the filename of the FileMarshaler?
> Handle clustering of outbound endpoints using "persisted sequence numbering"
> ----------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=41839#action_41839 ]
Guillaume Nodet commented on SM-1292:
-------------------------------------
If you use an incrementing counter in the file name, you don't really need a locking mechanism, because there will be no conflicts at all. For a distributed counter, I would think using a database is easier.
> Handle clustering of outbound endpoints using "persisted sequence numbering"
> ----------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=41893#action_41893 ]
Guillaume Nodet commented on SM-1292:
-------------------------------------
Yeah. A custom expression that would take a DataSource and using spring db layer (optionally) maybe to create a unique clustered incrementing counter.
> Handle clustering of outbound endpoints using "persisted sequence numbering"
> ----------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1292) Handle clustering of outbound endpoints
for append mode processing (using a distributed Lock Manager)
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ron Gavlin updated SM-1292:
---------------------------
Summary: Handle clustering of outbound endpoints for append mode processing (using a distributed Lock Manager) (was: Handle clustering of outbound endpoints using "persisted sequence numbering")
> Handle clustering of outbound endpoints for append mode processing (using a distributed Lock Manager)
> -----------------------------------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Posted by "Ron Gavlin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=41838#action_41838 ]
Ron Gavlin commented on SM-1292:
--------------------------------
Hi Guillaume,
Writing to the same file in append mode is another use case that I wasn't explicitly thinking about, but it is definitely a valid scenario.
In my use case, I have a follow-on system that expects an incrementing counter embedded in the filename that works across the cluster. The example described above seems to only handle the incrementing counter within a single vm. OpenESB supports using a control file whose contents is the current value of the counter. In order to implement such a facility, a distributed locking mechanism would be required as well as a convenient facility for storing the counter state across the cluster. If a control file was used for the distributed locking scheme, it would seem a convenient location to store the counter. This does however couple the locking facility with the state management. What are your thoughts?
> Handle clustering of outbound endpoints using "persisted sequence numbering"
> ----------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1292) Handle clustering of outbound endpoints
using "persisted sequence numbering"
Posted by "Guillaume Nodet (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=41837#action_41837 ]
Guillaume Nodet commented on SM-1292:
-------------------------------------
It can already be done easily afaik.
{code}
<file:sender service="test:service" endpoint="endpoint" directory="file:target/componentOutput">
<file:marshaler>
<sm:defaultFileMarshaler>
<sm:fileName>
<!-- lets use a header from the message -->
<sm:xpathString xpath="concat($name, '.xml')"/>
</sm:fileName>
</sm:defaultFileMarshaler>
</file:marshaler>
</file:sender>
{code}
Just implementing an expression that returns a unique identifier (or incremening counter) would work.
I think the problem of outbound clustering is precisely when several endpoints need to write to the same file (let's say you use an append mode somehow), and we need a locking mechanism.
Another problem is when a file provider write to a file, and a file consumer consume those files. Or even if you need to communicate to some external systems. In such cases, the consumer needs to know when the file has been fully written so that it can be read. This requires a locking mechanism. The file component uses a LockManager, but we have no distribtued implementation of it yet.
> Handle clustering of outbound endpoints using "persisted sequence numbering"
> ----------------------------------------------------------------------------
>
> Key: SM-1292
> URL: https://issues.apache.org/activemq/browse/SM-1292
> Project: ServiceMix
> Issue Type: New Feature
> Components: servicemix-file
> Affects Versions: 3.2.1
> Reporter: Ron Gavlin
>
> The user should be able to configure the output file name as a pattern including a "sequence number" (%{<seq_name>}) in the name, such that each output file when written out to the target folder has a unique file name that embeds the sequence number. The "persisted sequence number" is shared across the cluster. See the "Persisted Sequence Number" section of document http://wiki.open-esb.java.net/Wiki.jsp?page=FileBindingComponentClusteringSupportImplementation for a description of how Sun's OpenESB File Binding Component supports this requirement using a control file in the target directory.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.