[jira] [Commented] (APEXMALHAR-2434) JMSTransactionableStore uses Session.createQueue() which fails

["Sanjay M Pujare (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15898492#comment-15898492 ] 

Sanjay M Pujare commented on APEXMALHAR-2434:
---------------------------------------------

Going with the last option ie. using deterministic naming for the metadata queue, this is what we should do:

- for the actual application queue named "FOOBAR", the store should use the metadata queue "FOOBAR.metadata".
- all operators (partitions) writing to "FOOBAR" use message properties (javax.jms.Message.setIntProperty(String, int)) to set the operatorId with name "operatorId" on the message to be written.
- these operators use message selectors in createBrowser(queue) and createConsumers(queue) to only get messages with those respective message properties e.g. operator 2 only gets messages with message property operatorId==2.

This solution seems to be simple and elegant.

> JMSTransactionableStore uses Session.createQueue() which fails
> --------------------------------------------------------------
>
>                 Key: APEXMALHAR-2434
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2434
>             Project: Apache Apex Malhar
>          Issue Type: Bug
>            Reporter: Sanjay M Pujare
>
> JMSTransactionableStore needs to create a queue for storing metadata (lastWindowId etc) that will work across invocations to support fault tolerance and idempotency. 
> However as the createQueue Javadocs says: "Note that this method is not for creating the physical queue. The physical creation of queues is an administrative task and is not to be initiated by the JMS API." This causes a failure in actual tests with a production JMS based broker (such as IBM MQSeries). We will need to fix this in one of the following ways:
> - using an alternative store (HDFS or JDBC)
> - allow the user to specify a name for this metadata queue via a property
> - generate the name in a deterministic fashion from the subject of the queue
> The last 2 alternatives assume that the application user has created this metadata queue ahead of time from the admin console. We will need to document this in the malhar docs. The last alternative looks most attractive except if there are multiple JMS output operators (say partitions of an operator) writing to the same queue (subject) we will have to use some additional logic for them to share this single metadata queue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)