You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@synapse.apache.org by "Wright, Peter" <Pe...@six-group.com> on 2013/08/07 13:53:28 UTC
RE: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Hi Hiranja,
I have now implemented a class mediator that parses the (multiple)
syslog plain text messages and makes an XML payload from them.
I've also got an iterate mediator that splits the messages and
sends them individually as JMS messages to the messaging provider.
So far so good...
However, the individual JMS messages are now in XML form rather
than plain text (with all the XML headers that I don't want).
How can I get just the plain text payload to be sent?
Thanks
Peter
=====================
1. Input: Dummy (syslog) message sent to TCP listener (individual messages separated by NL)
20 This is msg number 1
20 This is msg number 2
20 This is msg number 3
2. SOAP message logged by synapse
<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">20 This is msg number 1
20 This is msg number 2
20 This is msg number 3
</text></soapenv:Body></soapenv:Envelope>
3. XML message returned by my class mediator
<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
4. JMS messages sent to the messaging provider
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 1</msg></soapenv:Body></soapenv:Envelope>
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 2</msg></soapenv:Body></soapenv:Envelope>
<?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 3</msg></soapenv:Body></soapenv:Envelope>
5. JMS messages that I want to be sent
This is msg number 1
This is msg number 2
This is msg number 3
6. synapse.xml
<definitions xmlns="http://ws.apache.org/ns/synapse">
<proxy name="proxyTcp2Jms" transports="tcp">
<target>
<inSequence>
<!-- Define TCP listener expects plain text (syslog) messages -->
<property name="messageType" value="text/plain" scope="axis2"/>
<property name="OUT_ONLY" value="true"/>
<property name="TRANSPORT_HEADERS" scope="axis2" action="remove"/>
<log level="full"/>
<!-- Split any multiple syslog messages for the iterator -->
<!-- and return as XML with child messages -->
<class name="com.sixtelekurs.classMediators.syslogMsgBuilder">
<log level="full"/>
</class>
<!-- Iterate over any multiple messages -->
<iterate id="syslogInterator" preservePayload="false" sequential="true" expression="//msg">
<log level="full"/>
<target>
<sequence>
<send>
<endpoint>
<address uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
</endpoint>
</send>
</sequence>
</target>
</iterate>
</inSequence>
<outSequence/>
<faultSequence>
<log level="full" category="ERROR" separator=","/>
</faultSequence>
</target>
<parameter name="transport.tcp.port">6060</parameter>
<parameter name="transport.tcp.contentType">text/plain</parameter>
</proxy>
</definitions>
-----Original Message-----
From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
Sent: Freitag, 26. Juli 2013 20:05
To: user@synapse.apache.org
Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
You should probably apply some kind of a content transformation to transform this payload into a XML payload with multiple "text" elements (one per each syslog message). But I'm not sure whether traditional techniques like XSLT and XQuery support that kind of transformations. After all you're trying to transform based on XML text and not the XML nodes. That's almost always very difficult.
IMO you're better off writing a custom mediator to do this transformation. Then you can use the iterate mediator to split the message into its constituents and send to the JMS queue. An even more elegant solution would be to write a custom message builder that knows how to parse the incoming batch requests and construct a proper XML payload out of it.
Thanks,
Hiranya
On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
> Hi,
>
> I got synapse working with a TCP-Listener accepting plain text messages
> as input, and the messages being sent as JMS messages to IBM Websphere MQ.
>
> Now we're in the test phase with "real" syslog (RFC-5424) data, and I'm having a problem
> because the TCP listener is receiving multiple syslog messages in one TCP message.
> The multiple messages are being sent on as one JMS message. The individual
> syslog messages are separated by NL and have the message length at the start.
>
> Here an example message from my log file with 8 syslog messages:
>
> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">
> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" BC="462" BK="842" Occured="1374760680" Reported="1374761340" Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"]
> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
> </text></soapenv:Body></soapenv:Envelope>
>
> What is the recommended way of handling this? Use the Iterate-Mediator to
> split the messages and send them individually? Do I have to use Xpath here,
> or is it possible to use e.g. javascript? Any examples would be appreciated.
>
> Thanks for any help,
> Peter
>
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this email immediately.
> Thank you.
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
The content of this e-mail is intended only for the confidential use of the person addressed.
If you are not the intended recipient, please notify the sender and delete this email immediately.
Thank you.
Re: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Posted by Andreas Veithen <an...@gmail.com>.
On Fri, Aug 9, 2013 at 9:26 PM, Rajika Kumarasiri
<ra...@gmail.com> wrote:
> Yeah, actually I have started working on a new syslog consumer
> implementation. As part of this I am also planning to write brand new TCP
> and UDP transports(for high volume syslog message deployments). I will
> report any progress as I go forward.
Why would the current UDP transport not be suitable for high volume messages?
Andreas
> Rajika
>
>
> On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hi...@gmail.com>wrote:
>
>>
>> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>>
>> > Hi Hiranja,
>> >
>> > Great, thanks a lot, now appears to be working correctly!!
>>
>> That's great.
>>
>> I think it would be good to have some syslog mediation support built into
>> Synapse. The correct solution is probably to implement a syslog message
>> builder and a formatter. There's a long standing jura issue regarding this
>> as well. We'll consider this feature for a future release.
>>
>> Thanks,
>> Hiranya
>>
>> > I changed my class mediator to generate the XML as you suggested, and
>> then had to
>> > include the namespace in the iterate definition and include it in the
>> xpath expression
>> > before it worked correctly.
>> >
>> > Best regards,
>> > Peter
>> >
>> > =================================
>> >
>> > 1. XML message returned by my class mediator
>> >
>> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
>> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
>> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg
>> number 3</text></soapenv:Body></soapenv:Envelope>
>> >
>> > 2. synapse.xml fragment (iterator definition)
>> >
>> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
>> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
>> > <target>
>> > <sequence>
>> > <send>
>> > <endpoint>
>> > <address .../>
>> > </endpoint>
>> > </send>
>> > </sequence>
>> > </target>
>> > </iterate>
>> >
>> > -----Original Message-----
>> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> > Sent: Freitag, 9. August 2013 01:54
>> > To: user@synapse.apache.org
>> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
>> in one TCP message?
>> >
>> >
>> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>> >
>> >> Hi Hiranja,
>> >>
>> >> I have now implemented a class mediator that parses the (multiple)
>> >> syslog plain text messages and makes an XML payload from them.
>> >> I've also got an iterate mediator that splits the messages and
>> >> sends them individually as JMS messages to the messaging provider.
>> >> So far so good...
>> >>
>> >> However, the individual JMS messages are now in XML form rather
>> >> than plain text (with all the XML headers that I don't want).
>> >> How can I get just the plain text payload to be sent?
>> >>
>> >> Thanks
>> >> Peter
>> >>
>> >> =====================
>> >>
>> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
>> messages separated by NL)
>> >> 20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >>
>> >> 2. SOAP message logged by synapse
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >> </text></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 3. XML message returned by my class mediator
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
>> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number
>> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>> >
>> > This should create a message with multiple <text> elements. See the
>> output from step 2 above. It has a single <text> element with all the
>> content. You should break it down to multiple <text> elements. The plain
>> text serializer of Axis2 looks for this specific <text> element when
>> sending plain text messages.
>> >
>> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
>> of the <text> element too. It's also required.
>> >
>> > Thanks,
>> > Hiranya
>> >
>> >>
>> >> 4. JMS messages sent to the messaging provider
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 1</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 2</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 3</msg></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 5. JMS messages that I want to be sent
>> >> This is msg number 1
>> >> This is msg number 2
>> >> This is msg number 3
>> >>
>> >> 6. synapse.xml
>> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> >> <proxy name="proxyTcp2Jms" transports="tcp">
>> >> <target>
>> >> <inSequence>
>> >> <!-- Define TCP listener expects plain text (syslog)
>> messages -->
>> >> <property name="messageType" value="text/plain"
>> scope="axis2"/>
>> >> <property name="OUT_ONLY" value="true"/>
>> >> <property name="TRANSPORT_HEADERS" scope="axis2"
>> action="remove"/>
>> >> <log level="full"/>
>> >>
>> >> <!-- Split any multiple syslog messages for the iterator
>> -->
>> >> <!-- and return as XML with child messages
>> -->
>> >> <class
>> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> >> <log level="full"/>
>> >> </class>
>> >>
>> >> <!-- Iterate over any multiple messages -->
>> >> <iterate id="syslogInterator" preservePayload="false"
>> sequential="true" expression="//msg">
>> >> <log level="full"/>
>> >> <target>
>> >> <sequence>
>> >> <send>
>> >> <endpoint>
>> >> <address
>> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
>> >> </endpoint>
>> >> </send>
>> >> </sequence>
>> >> </target>
>> >> </iterate>
>> >> </inSequence>
>> >> <outSequence/>
>> >> <faultSequence>
>> >> <log level="full" category="ERROR" separator=","/>
>> >> </faultSequence>
>> >> </target>
>> >> <parameter name="transport.tcp.port">6060</parameter>
>> >> <parameter name="transport.tcp.contentType">text/plain</parameter>
>> >> </proxy>
>> >>
>> >> </definitions>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> >> Sent: Freitag, 26. Juli 2013 20:05
>> >> To: user@synapse.apache.org
>> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
>> messages in one TCP message?
>> >>
>> >> You should probably apply some kind of a content transformation to
>> transform this payload into a XML payload with multiple "text" elements
>> (one per each syslog message). But I'm not sure whether traditional
>> techniques like XSLT and XQuery support that kind of transformations. After
>> all you're trying to transform based on XML text and not the XML nodes.
>> That's almost always very difficult.
>> >>
>> >> IMO you're better off writing a custom mediator to do this
>> transformation. Then you can use the iterate mediator to split the message
>> into its constituents and send to the JMS queue. An even more elegant
>> solution would be to write a custom message builder that knows how to parse
>> the incoming batch requests and construct a proper XML payload out of it.
>> >>
>> >> Thanks,
>> >> Hiranya
>> >>
>> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
>> Peter.Wright@six-group.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I got synapse working with a TCP-Listener accepting plain text messages
>> >>> as input, and the messages being sent as JMS messages to IBM Websphere
>> MQ.
>> >>>
>> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
>> I'm having a problem
>> >>> because the TCP listener is receiving multiple syslog messages in one
>> TCP message.
>> >>> The multiple messages are being sent on as one JMS message. The
>> individual
>> >>> syslog messages are separated by NL and have the message length at the
>> start.
>> >>>
>> >>> Here an example message from my log file with 8 syslog messages:
>> >>>
>> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: ,
>> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
>> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">
>> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
>> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
>> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
>> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
>> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 -
>> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
>> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
>> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
>> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
>> on mdzhldos02)"]
>> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> >>> </text></soapenv:Body></soapenv:Envelope>
>> >>>
>> >>> What is the recommended way of handling this? Use the Iterate-Mediator
>> to
>> >>> split the messages and send them individually? Do I have to use Xpath
>> here,
>> >>> or is it possible to use e.g. javascript? Any examples would be
>> appreciated.
>> >>>
>> >>> Thanks for any help,
>> >>> Peter
>> >>>
>> >>>
>> >>> The content of this e-mail is intended only for the confidential use
>> of the person addressed.
>> >>> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >>> Thank you.
>> >>
>> >> --
>> >> Hiranya Jayathilaka
>> >> Mayhem Lab/RACE Lab;
>> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> >> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> >> Blog: http://techfeast-hiranya.blogspot.com
>> >>
>> >> The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> >> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >> Thank you.
>> >
>> > --
>> > Hiranya Jayathilaka
>> > Mayhem Lab/RACE Lab;
>> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> > Blog: http://techfeast-hiranya.blogspot.com
>> >
>> > The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> > If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> > Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org
Re: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Posted by Andreas Veithen <an...@gmail.com>.
On Fri, Aug 9, 2013 at 9:26 PM, Rajika Kumarasiri
<ra...@gmail.com> wrote:
> Yeah, actually I have started working on a new syslog consumer
> implementation. As part of this I am also planning to write brand new TCP
> and UDP transports(for high volume syslog message deployments). I will
> report any progress as I go forward.
Why would the current UDP transport not be suitable for high volume messages?
Andreas
> Rajika
>
>
> On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hi...@gmail.com>wrote:
>
>>
>> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>>
>> > Hi Hiranja,
>> >
>> > Great, thanks a lot, now appears to be working correctly!!
>>
>> That's great.
>>
>> I think it would be good to have some syslog mediation support built into
>> Synapse. The correct solution is probably to implement a syslog message
>> builder and a formatter. There's a long standing jura issue regarding this
>> as well. We'll consider this feature for a future release.
>>
>> Thanks,
>> Hiranya
>>
>> > I changed my class mediator to generate the XML as you suggested, and
>> then had to
>> > include the namespace in the iterate definition and include it in the
>> xpath expression
>> > before it worked correctly.
>> >
>> > Best regards,
>> > Peter
>> >
>> > =================================
>> >
>> > 1. XML message returned by my class mediator
>> >
>> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
>> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
>> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg
>> number 3</text></soapenv:Body></soapenv:Envelope>
>> >
>> > 2. synapse.xml fragment (iterator definition)
>> >
>> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
>> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
>> > <target>
>> > <sequence>
>> > <send>
>> > <endpoint>
>> > <address .../>
>> > </endpoint>
>> > </send>
>> > </sequence>
>> > </target>
>> > </iterate>
>> >
>> > -----Original Message-----
>> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> > Sent: Freitag, 9. August 2013 01:54
>> > To: user@synapse.apache.org
>> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
>> in one TCP message?
>> >
>> >
>> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>> >
>> >> Hi Hiranja,
>> >>
>> >> I have now implemented a class mediator that parses the (multiple)
>> >> syslog plain text messages and makes an XML payload from them.
>> >> I've also got an iterate mediator that splits the messages and
>> >> sends them individually as JMS messages to the messaging provider.
>> >> So far so good...
>> >>
>> >> However, the individual JMS messages are now in XML form rather
>> >> than plain text (with all the XML headers that I don't want).
>> >> How can I get just the plain text payload to be sent?
>> >>
>> >> Thanks
>> >> Peter
>> >>
>> >> =====================
>> >>
>> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
>> messages separated by NL)
>> >> 20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >>
>> >> 2. SOAP message logged by synapse
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >> </text></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 3. XML message returned by my class mediator
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
>> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number
>> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>> >
>> > This should create a message with multiple <text> elements. See the
>> output from step 2 above. It has a single <text> element with all the
>> content. You should break it down to multiple <text> elements. The plain
>> text serializer of Axis2 looks for this specific <text> element when
>> sending plain text messages.
>> >
>> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
>> of the <text> element too. It's also required.
>> >
>> > Thanks,
>> > Hiranya
>> >
>> >>
>> >> 4. JMS messages sent to the messaging provider
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 1</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 2</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 3</msg></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 5. JMS messages that I want to be sent
>> >> This is msg number 1
>> >> This is msg number 2
>> >> This is msg number 3
>> >>
>> >> 6. synapse.xml
>> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> >> <proxy name="proxyTcp2Jms" transports="tcp">
>> >> <target>
>> >> <inSequence>
>> >> <!-- Define TCP listener expects plain text (syslog)
>> messages -->
>> >> <property name="messageType" value="text/plain"
>> scope="axis2"/>
>> >> <property name="OUT_ONLY" value="true"/>
>> >> <property name="TRANSPORT_HEADERS" scope="axis2"
>> action="remove"/>
>> >> <log level="full"/>
>> >>
>> >> <!-- Split any multiple syslog messages for the iterator
>> -->
>> >> <!-- and return as XML with child messages
>> -->
>> >> <class
>> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> >> <log level="full"/>
>> >> </class>
>> >>
>> >> <!-- Iterate over any multiple messages -->
>> >> <iterate id="syslogInterator" preservePayload="false"
>> sequential="true" expression="//msg">
>> >> <log level="full"/>
>> >> <target>
>> >> <sequence>
>> >> <send>
>> >> <endpoint>
>> >> <address
>> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
>> >> </endpoint>
>> >> </send>
>> >> </sequence>
>> >> </target>
>> >> </iterate>
>> >> </inSequence>
>> >> <outSequence/>
>> >> <faultSequence>
>> >> <log level="full" category="ERROR" separator=","/>
>> >> </faultSequence>
>> >> </target>
>> >> <parameter name="transport.tcp.port">6060</parameter>
>> >> <parameter name="transport.tcp.contentType">text/plain</parameter>
>> >> </proxy>
>> >>
>> >> </definitions>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> >> Sent: Freitag, 26. Juli 2013 20:05
>> >> To: user@synapse.apache.org
>> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
>> messages in one TCP message?
>> >>
>> >> You should probably apply some kind of a content transformation to
>> transform this payload into a XML payload with multiple "text" elements
>> (one per each syslog message). But I'm not sure whether traditional
>> techniques like XSLT and XQuery support that kind of transformations. After
>> all you're trying to transform based on XML text and not the XML nodes.
>> That's almost always very difficult.
>> >>
>> >> IMO you're better off writing a custom mediator to do this
>> transformation. Then you can use the iterate mediator to split the message
>> into its constituents and send to the JMS queue. An even more elegant
>> solution would be to write a custom message builder that knows how to parse
>> the incoming batch requests and construct a proper XML payload out of it.
>> >>
>> >> Thanks,
>> >> Hiranya
>> >>
>> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
>> Peter.Wright@six-group.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I got synapse working with a TCP-Listener accepting plain text messages
>> >>> as input, and the messages being sent as JMS messages to IBM Websphere
>> MQ.
>> >>>
>> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
>> I'm having a problem
>> >>> because the TCP listener is receiving multiple syslog messages in one
>> TCP message.
>> >>> The multiple messages are being sent on as one JMS message. The
>> individual
>> >>> syslog messages are separated by NL and have the message length at the
>> start.
>> >>>
>> >>> Here an example message from my log file with 8 syslog messages:
>> >>>
>> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: ,
>> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
>> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">
>> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
>> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
>> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
>> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
>> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 -
>> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
>> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
>> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
>> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
>> on mdzhldos02)"]
>> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> >>> </text></soapenv:Body></soapenv:Envelope>
>> >>>
>> >>> What is the recommended way of handling this? Use the Iterate-Mediator
>> to
>> >>> split the messages and send them individually? Do I have to use Xpath
>> here,
>> >>> or is it possible to use e.g. javascript? Any examples would be
>> appreciated.
>> >>>
>> >>> Thanks for any help,
>> >>> Peter
>> >>>
>> >>>
>> >>> The content of this e-mail is intended only for the confidential use
>> of the person addressed.
>> >>> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >>> Thank you.
>> >>
>> >> --
>> >> Hiranya Jayathilaka
>> >> Mayhem Lab/RACE Lab;
>> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> >> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> >> Blog: http://techfeast-hiranya.blogspot.com
>> >>
>> >> The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> >> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >> Thank you.
>> >
>> > --
>> > Hiranya Jayathilaka
>> > Mayhem Lab/RACE Lab;
>> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> > Blog: http://techfeast-hiranya.blogspot.com
>> >
>> > The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> > If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> > Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>
Re: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Posted by Rajika Kumarasiri <ra...@gmail.com>.
Yeah, actually I have started working on a new syslog consumer
implementation. As part of this I am also planning to write brand new TCP
and UDP transports(for high volume syslog message deployments). I will
report any progress as I go forward.
Rajika
On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hi...@gmail.com>wrote:
>
> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com>
> wrote:
>
> > Hi Hiranja,
> >
> > Great, thanks a lot, now appears to be working correctly!!
>
> That's great.
>
> I think it would be good to have some syslog mediation support built into
> Synapse. The correct solution is probably to implement a syslog message
> builder and a formatter. There's a long standing jura issue regarding this
> as well. We'll consider this feature for a future release.
>
> Thanks,
> Hiranya
>
> > I changed my class mediator to generate the XML as you suggested, and
> then had to
> > include the namespace in the iterate definition and include it in the
> xpath expression
> > before it worked correctly.
> >
> > Best regards,
> > Peter
> >
> > =================================
> >
> > 1. XML message returned by my class mediator
> >
> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg
> number 3</text></soapenv:Body></soapenv:Envelope>
> >
> > 2. synapse.xml fragment (iterator definition)
> >
> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
> > <target>
> > <sequence>
> > <send>
> > <endpoint>
> > <address .../>
> > </endpoint>
> > </send>
> > </sequence>
> > </target>
> > </iterate>
> >
> > -----Original Message-----
> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> > Sent: Freitag, 9. August 2013 01:54
> > To: user@synapse.apache.org
> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
> in one TCP message?
> >
> >
> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com>
> wrote:
> >
> >> Hi Hiranja,
> >>
> >> I have now implemented a class mediator that parses the (multiple)
> >> syslog plain text messages and makes an XML payload from them.
> >> I've also got an iterate mediator that splits the messages and
> >> sends them individually as JMS messages to the messaging provider.
> >> So far so good...
> >>
> >> However, the individual JMS messages are now in XML form rather
> >> than plain text (with all the XML headers that I don't want).
> >> How can I get just the plain text payload to be sent?
> >>
> >> Thanks
> >> Peter
> >>
> >> =====================
> >>
> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
> messages separated by NL)
> >> 20 This is msg number 1
> >> 20 This is msg number 2
> >> 20 This is msg number 3
> >>
> >> 2. SOAP message logged by synapse
> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
> >> 20 This is msg number 2
> >> 20 This is msg number 3
> >> </text></soapenv:Body></soapenv:Envelope>
> >>
> >> 3. XML message returned by my class mediator
> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number
> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
> >
> > This should create a message with multiple <text> elements. See the
> output from step 2 above. It has a single <text> element with all the
> content. You should break it down to multiple <text> elements. The plain
> text serializer of Axis2 looks for this specific <text> element when
> sending plain text messages.
> >
> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
> of the <text> element too. It's also required.
> >
> > Thanks,
> > Hiranya
> >
> >>
> >> 4. JMS messages sent to the messaging provider
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 1</msg></soapenv:Body></soapenv:Envelope>
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 2</msg></soapenv:Body></soapenv:Envelope>
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 3</msg></soapenv:Body></soapenv:Envelope>
> >>
> >> 5. JMS messages that I want to be sent
> >> This is msg number 1
> >> This is msg number 2
> >> This is msg number 3
> >>
> >> 6. synapse.xml
> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
> >> <proxy name="proxyTcp2Jms" transports="tcp">
> >> <target>
> >> <inSequence>
> >> <!-- Define TCP listener expects plain text (syslog)
> messages -->
> >> <property name="messageType" value="text/plain"
> scope="axis2"/>
> >> <property name="OUT_ONLY" value="true"/>
> >> <property name="TRANSPORT_HEADERS" scope="axis2"
> action="remove"/>
> >> <log level="full"/>
> >>
> >> <!-- Split any multiple syslog messages for the iterator
> -->
> >> <!-- and return as XML with child messages
> -->
> >> <class
> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
> >> <log level="full"/>
> >> </class>
> >>
> >> <!-- Iterate over any multiple messages -->
> >> <iterate id="syslogInterator" preservePayload="false"
> sequential="true" expression="//msg">
> >> <log level="full"/>
> >> <target>
> >> <sequence>
> >> <send>
> >> <endpoint>
> >> <address
> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
> >> </endpoint>
> >> </send>
> >> </sequence>
> >> </target>
> >> </iterate>
> >> </inSequence>
> >> <outSequence/>
> >> <faultSequence>
> >> <log level="full" category="ERROR" separator=","/>
> >> </faultSequence>
> >> </target>
> >> <parameter name="transport.tcp.port">6060</parameter>
> >> <parameter name="transport.tcp.contentType">text/plain</parameter>
> >> </proxy>
> >>
> >> </definitions>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> >> Sent: Freitag, 26. Juli 2013 20:05
> >> To: user@synapse.apache.org
> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
> messages in one TCP message?
> >>
> >> You should probably apply some kind of a content transformation to
> transform this payload into a XML payload with multiple "text" elements
> (one per each syslog message). But I'm not sure whether traditional
> techniques like XSLT and XQuery support that kind of transformations. After
> all you're trying to transform based on XML text and not the XML nodes.
> That's almost always very difficult.
> >>
> >> IMO you're better off writing a custom mediator to do this
> transformation. Then you can use the iterate mediator to split the message
> into its constituents and send to the JMS queue. An even more elegant
> solution would be to write a custom message builder that knows how to parse
> the incoming batch requests and construct a proper XML payload out of it.
> >>
> >> Thanks,
> >> Hiranya
> >>
> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
> Peter.Wright@six-group.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I got synapse working with a TCP-Listener accepting plain text messages
> >>> as input, and the messages being sent as JMS messages to IBM Websphere
> MQ.
> >>>
> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
> I'm having a problem
> >>> because the TCP listener is receiving multiple syslog messages in one
> TCP message.
> >>> The multiple messages are being sent on as one JMS message. The
> individual
> >>> syslog messages are separated by NL and have the message length at the
> start.
> >>>
> >>> Here an example message from my log file with 8 syslog messages:
> >>>
> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: ,
> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">
> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 -
> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
> on mdzhldos02)"]
> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
> >>> </text></soapenv:Body></soapenv:Envelope>
> >>>
> >>> What is the recommended way of handling this? Use the Iterate-Mediator
> to
> >>> split the messages and send them individually? Do I have to use Xpath
> here,
> >>> or is it possible to use e.g. javascript? Any examples would be
> appreciated.
> >>>
> >>> Thanks for any help,
> >>> Peter
> >>>
> >>>
> >>> The content of this e-mail is intended only for the confidential use
> of the person addressed.
> >>> If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> >>> Thank you.
> >>
> >> --
> >> Hiranya Jayathilaka
> >> Mayhem Lab/RACE Lab;
> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> >> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> >> Blog: http://techfeast-hiranya.blogspot.com
> >>
> >> The content of this e-mail is intended only for the confidential use of
> the person addressed.
> >> If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> >> Thank you.
> >
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> > The content of this e-mail is intended only for the confidential use of
> the person addressed.
> > If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> > Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
Re: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Posted by Rajika Kumarasiri <ra...@gmail.com>.
Yeah, actually I have started working on a new syslog consumer
implementation. As part of this I am also planning to write brand new TCP
and UDP transports(for high volume syslog message deployments). I will
report any progress as I go forward.
Rajika
On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hi...@gmail.com>wrote:
>
> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com>
> wrote:
>
> > Hi Hiranja,
> >
> > Great, thanks a lot, now appears to be working correctly!!
>
> That's great.
>
> I think it would be good to have some syslog mediation support built into
> Synapse. The correct solution is probably to implement a syslog message
> builder and a formatter. There's a long standing jura issue regarding this
> as well. We'll consider this feature for a future release.
>
> Thanks,
> Hiranya
>
> > I changed my class mediator to generate the XML as you suggested, and
> then had to
> > include the namespace in the iterate definition and include it in the
> xpath expression
> > before it worked correctly.
> >
> > Best regards,
> > Peter
> >
> > =================================
> >
> > 1. XML message returned by my class mediator
> >
> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg
> number 3</text></soapenv:Body></soapenv:Envelope>
> >
> > 2. synapse.xml fragment (iterator definition)
> >
> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
> > <target>
> > <sequence>
> > <send>
> > <endpoint>
> > <address .../>
> > </endpoint>
> > </send>
> > </sequence>
> > </target>
> > </iterate>
> >
> > -----Original Message-----
> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> > Sent: Freitag, 9. August 2013 01:54
> > To: user@synapse.apache.org
> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
> in one TCP message?
> >
> >
> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com>
> wrote:
> >
> >> Hi Hiranja,
> >>
> >> I have now implemented a class mediator that parses the (multiple)
> >> syslog plain text messages and makes an XML payload from them.
> >> I've also got an iterate mediator that splits the messages and
> >> sends them individually as JMS messages to the messaging provider.
> >> So far so good...
> >>
> >> However, the individual JMS messages are now in XML form rather
> >> than plain text (with all the XML headers that I don't want).
> >> How can I get just the plain text payload to be sent?
> >>
> >> Thanks
> >> Peter
> >>
> >> =====================
> >>
> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
> messages separated by NL)
> >> 20 This is msg number 1
> >> 20 This is msg number 2
> >> 20 This is msg number 3
> >>
> >> 2. SOAP message logged by synapse
> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
> >> 20 This is msg number 2
> >> 20 This is msg number 3
> >> </text></soapenv:Body></soapenv:Envelope>
> >>
> >> 3. XML message returned by my class mediator
> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number
> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
> >
> > This should create a message with multiple <text> elements. See the
> output from step 2 above. It has a single <text> element with all the
> content. You should break it down to multiple <text> elements. The plain
> text serializer of Axis2 looks for this specific <text> element when
> sending plain text messages.
> >
> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
> of the <text> element too. It's also required.
> >
> > Thanks,
> > Hiranya
> >
> >>
> >> 4. JMS messages sent to the messaging provider
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 1</msg></soapenv:Body></soapenv:Envelope>
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 2</msg></soapenv:Body></soapenv:Envelope>
> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
> number 3</msg></soapenv:Body></soapenv:Envelope>
> >>
> >> 5. JMS messages that I want to be sent
> >> This is msg number 1
> >> This is msg number 2
> >> This is msg number 3
> >>
> >> 6. synapse.xml
> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
> >> <proxy name="proxyTcp2Jms" transports="tcp">
> >> <target>
> >> <inSequence>
> >> <!-- Define TCP listener expects plain text (syslog)
> messages -->
> >> <property name="messageType" value="text/plain"
> scope="axis2"/>
> >> <property name="OUT_ONLY" value="true"/>
> >> <property name="TRANSPORT_HEADERS" scope="axis2"
> action="remove"/>
> >> <log level="full"/>
> >>
> >> <!-- Split any multiple syslog messages for the iterator
> -->
> >> <!-- and return as XML with child messages
> -->
> >> <class
> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
> >> <log level="full"/>
> >> </class>
> >>
> >> <!-- Iterate over any multiple messages -->
> >> <iterate id="syslogInterator" preservePayload="false"
> sequential="true" expression="//msg">
> >> <log level="full"/>
> >> <target>
> >> <sequence>
> >> <send>
> >> <endpoint>
> >> <address
> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
> >> </endpoint>
> >> </send>
> >> </sequence>
> >> </target>
> >> </iterate>
> >> </inSequence>
> >> <outSequence/>
> >> <faultSequence>
> >> <log level="full" category="ERROR" separator=","/>
> >> </faultSequence>
> >> </target>
> >> <parameter name="transport.tcp.port">6060</parameter>
> >> <parameter name="transport.tcp.contentType">text/plain</parameter>
> >> </proxy>
> >>
> >> </definitions>
> >>
> >>
> >>
> >> -----Original Message-----
> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> >> Sent: Freitag, 26. Juli 2013 20:05
> >> To: user@synapse.apache.org
> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
> messages in one TCP message?
> >>
> >> You should probably apply some kind of a content transformation to
> transform this payload into a XML payload with multiple "text" elements
> (one per each syslog message). But I'm not sure whether traditional
> techniques like XSLT and XQuery support that kind of transformations. After
> all you're trying to transform based on XML text and not the XML nodes.
> That's almost always very difficult.
> >>
> >> IMO you're better off writing a custom mediator to do this
> transformation. Then you can use the iterate mediator to split the message
> into its constituents and send to the JMS queue. An even more elegant
> solution would be to write a custom message builder that knows how to parse
> the incoming batch requests and construct a proper XML payload out of it.
> >>
> >> Thanks,
> >> Hiranya
> >>
> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
> Peter.Wright@six-group.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> I got synapse working with a TCP-Listener accepting plain text messages
> >>> as input, and the messages being sent as JMS messages to IBM Websphere
> MQ.
> >>>
> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
> I'm having a problem
> >>> because the TCP listener is receiving multiple syslog messages in one
> TCP message.
> >>> The multiple messages are being sent on as one JMS message. The
> individual
> >>> syslog messages are separated by NL and have the message length at the
> start.
> >>>
> >>> Here an example message from my log file with 8 syslog messages:
> >>>
> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: ,
> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
> http://ws.apache.org/commons/ns/payload">
> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 -
> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
> on mdzhldos02)"]
> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
> >>> </text></soapenv:Body></soapenv:Envelope>
> >>>
> >>> What is the recommended way of handling this? Use the Iterate-Mediator
> to
> >>> split the messages and send them individually? Do I have to use Xpath
> here,
> >>> or is it possible to use e.g. javascript? Any examples would be
> appreciated.
> >>>
> >>> Thanks for any help,
> >>> Peter
> >>>
> >>>
> >>> The content of this e-mail is intended only for the confidential use
> of the person addressed.
> >>> If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> >>> Thank you.
> >>
> >> --
> >> Hiranya Jayathilaka
> >> Mayhem Lab/RACE Lab;
> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> >> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> >> Blog: http://techfeast-hiranya.blogspot.com
> >>
> >> The content of this e-mail is intended only for the confidential use of
> the person addressed.
> >> If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> >> Thank you.
> >
> > --
> > Hiranya Jayathilaka
> > Mayhem Lab/RACE Lab;
> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> > Blog: http://techfeast-hiranya.blogspot.com
> >
> > The content of this e-mail is intended only for the confidential use of
> the person addressed.
> > If you are not the intended recipient, please notify the sender and
> delete this email immediately.
> > Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
>
Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
> Hi Hiranja,
>
> Great, thanks a lot, now appears to be working correctly!!
That's great.
I think it would be good to have some syslog mediation support built into Synapse. The correct solution is probably to implement a syslog message builder and a formatter. There's a long standing jura issue regarding this as well. We'll consider this feature for a future release.
Thanks,
Hiranya
> I changed my class mediator to generate the XML as you suggested, and then had to
> include the namespace in the iterate definition and include it in the xpath expression
> before it worked correctly.
>
> Best regards,
> Peter
>
> =================================
>
> 1. XML message returned by my class mediator
>
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 3</text></soapenv:Body></soapenv:Envelope>
>
> 2. synapse.xml fragment (iterator definition)
>
> <iterate id="syslogInterator" preservePayload="false" sequential="true" xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
> <target>
> <sequence>
> <send>
> <endpoint>
> <address .../>
> </endpoint>
> </send>
> </sequence>
> </target>
> </iterate>
>
> -----Original Message-----
> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> Sent: Freitag, 9. August 2013 01:54
> To: user@synapse.apache.org
> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>
>
> On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>
>> Hi Hiranja,
>>
>> I have now implemented a class mediator that parses the (multiple)
>> syslog plain text messages and makes an XML payload from them.
>> I've also got an iterate mediator that splits the messages and
>> sends them individually as JMS messages to the messaging provider.
>> So far so good...
>>
>> However, the individual JMS messages are now in XML form rather
>> than plain text (with all the XML headers that I don't want).
>> How can I get just the plain text payload to be sent?
>>
>> Thanks
>> Peter
>>
>> =====================
>>
>> 1. Input: Dummy (syslog) message sent to TCP listener (individual messages separated by NL)
>> 20 This is msg number 1
>> 20 This is msg number 2
>> 20 This is msg number 3
>>
>> 2. SOAP message logged by synapse
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> 20 This is msg number 2
>> 20 This is msg number 3
>> </text></soapenv:Body></soapenv:Envelope>
>>
>> 3. XML message returned by my class mediator
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>
> This should create a message with multiple <text> elements. See the output from step 2 above. It has a single <text> element with all the content. You should break it down to multiple <text> elements. The plain text serializer of Axis2 looks for this specific <text> element when sending plain text messages.
>
> Pay attention to the namespace (http://ws.apache.org/commons/ns/payload) of the <text> element too. It's also required.
>
> Thanks,
> Hiranya
>
>>
>> 4. JMS messages sent to the messaging provider
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 1</msg></soapenv:Body></soapenv:Envelope>
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 2</msg></soapenv:Body></soapenv:Envelope>
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 3</msg></soapenv:Body></soapenv:Envelope>
>>
>> 5. JMS messages that I want to be sent
>> This is msg number 1
>> This is msg number 2
>> This is msg number 3
>>
>> 6. synapse.xml
>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> <proxy name="proxyTcp2Jms" transports="tcp">
>> <target>
>> <inSequence>
>> <!-- Define TCP listener expects plain text (syslog) messages -->
>> <property name="messageType" value="text/plain" scope="axis2"/>
>> <property name="OUT_ONLY" value="true"/>
>> <property name="TRANSPORT_HEADERS" scope="axis2" action="remove"/>
>> <log level="full"/>
>>
>> <!-- Split any multiple syslog messages for the iterator -->
>> <!-- and return as XML with child messages -->
>> <class name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> <log level="full"/>
>> </class>
>>
>> <!-- Iterate over any multiple messages -->
>> <iterate id="syslogInterator" preservePayload="false" sequential="true" expression="//msg">
>> <log level="full"/>
>> <target>
>> <sequence>
>> <send>
>> <endpoint>
>> <address uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
>> </endpoint>
>> </send>
>> </sequence>
>> </target>
>> </iterate>
>> </inSequence>
>> <outSequence/>
>> <faultSequence>
>> <log level="full" category="ERROR" separator=","/>
>> </faultSequence>
>> </target>
>> <parameter name="transport.tcp.port">6060</parameter>
>> <parameter name="transport.tcp.contentType">text/plain</parameter>
>> </proxy>
>>
>> </definitions>
>>
>>
>>
>> -----Original Message-----
>> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> Sent: Freitag, 26. Juli 2013 20:05
>> To: user@synapse.apache.org
>> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>>
>> You should probably apply some kind of a content transformation to transform this payload into a XML payload with multiple "text" elements (one per each syslog message). But I'm not sure whether traditional techniques like XSLT and XQuery support that kind of transformations. After all you're trying to transform based on XML text and not the XML nodes. That's almost always very difficult.
>>
>> IMO you're better off writing a custom mediator to do this transformation. Then you can use the iterate mediator to split the message into its constituents and send to the JMS queue. An even more elegant solution would be to write a custom message builder that knows how to parse the incoming batch requests and construct a proper XML payload out of it.
>>
>> Thanks,
>> Hiranya
>>
>> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>>
>>> Hi,
>>>
>>> I got synapse working with a TCP-Listener accepting plain text messages
>>> as input, and the messages being sent as JMS messages to IBM Websphere MQ.
>>>
>>> Now we're in the test phase with "real" syslog (RFC-5424) data, and I'm having a problem
>>> because the TCP listener is receiving multiple syslog messages in one TCP message.
>>> The multiple messages are being sent on as one JMS message. The individual
>>> syslog messages are separated by NL and have the message length at the start.
>>>
>>> Here an example message from my log file with 8 syslog messages:
>>>
>>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">
>>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" BC="462" BK="842" Occured="1374760680" Reported="1374761340" Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"]
>>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>>> </text></soapenv:Body></soapenv:Envelope>
>>>
>>> What is the recommended way of handling this? Use the Iterate-Mediator to
>>> split the messages and send them individually? Do I have to use Xpath here,
>>> or is it possible to use e.g. javascript? Any examples would be appreciated.
>>>
>>> Thanks for any help,
>>> Peter
>>>
>>>
>>> The content of this e-mail is intended only for the confidential use of the person addressed.
>>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>>> Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>> The content of this e-mail is intended only for the confidential use of the person addressed.
>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>> Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this email immediately.
> Thank you.
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
> Hi Hiranja,
>
> Great, thanks a lot, now appears to be working correctly!!
That's great.
I think it would be good to have some syslog mediation support built into Synapse. The correct solution is probably to implement a syslog message builder and a formatter. There's a long standing jura issue regarding this as well. We'll consider this feature for a future release.
Thanks,
Hiranya
> I changed my class mediator to generate the XML as you suggested, and then had to
> include the namespace in the iterate definition and include it in the xpath expression
> before it worked correctly.
>
> Best regards,
> Peter
>
> =================================
>
> 1. XML message returned by my class mediator
>
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 3</text></soapenv:Body></soapenv:Envelope>
>
> 2. synapse.xml fragment (iterator definition)
>
> <iterate id="syslogInterator" preservePayload="false" sequential="true" xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
> <target>
> <sequence>
> <send>
> <endpoint>
> <address .../>
> </endpoint>
> </send>
> </sequence>
> </target>
> </iterate>
>
> -----Original Message-----
> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> Sent: Freitag, 9. August 2013 01:54
> To: user@synapse.apache.org
> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>
>
> On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>
>> Hi Hiranja,
>>
>> I have now implemented a class mediator that parses the (multiple)
>> syslog plain text messages and makes an XML payload from them.
>> I've also got an iterate mediator that splits the messages and
>> sends them individually as JMS messages to the messaging provider.
>> So far so good...
>>
>> However, the individual JMS messages are now in XML form rather
>> than plain text (with all the XML headers that I don't want).
>> How can I get just the plain text payload to be sent?
>>
>> Thanks
>> Peter
>>
>> =====================
>>
>> 1. Input: Dummy (syslog) message sent to TCP listener (individual messages separated by NL)
>> 20 This is msg number 1
>> 20 This is msg number 2
>> 20 This is msg number 3
>>
>> 2. SOAP message logged by synapse
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> 20 This is msg number 2
>> 20 This is msg number 3
>> </text></soapenv:Body></soapenv:Envelope>
>>
>> 3. XML message returned by my class mediator
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>
> This should create a message with multiple <text> elements. See the output from step 2 above. It has a single <text> element with all the content. You should break it down to multiple <text> elements. The plain text serializer of Axis2 looks for this specific <text> element when sending plain text messages.
>
> Pay attention to the namespace (http://ws.apache.org/commons/ns/payload) of the <text> element too. It's also required.
>
> Thanks,
> Hiranya
>
>>
>> 4. JMS messages sent to the messaging provider
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 1</msg></soapenv:Body></soapenv:Envelope>
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 2</msg></soapenv:Body></soapenv:Envelope>
>> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 3</msg></soapenv:Body></soapenv:Envelope>
>>
>> 5. JMS messages that I want to be sent
>> This is msg number 1
>> This is msg number 2
>> This is msg number 3
>>
>> 6. synapse.xml
>> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> <proxy name="proxyTcp2Jms" transports="tcp">
>> <target>
>> <inSequence>
>> <!-- Define TCP listener expects plain text (syslog) messages -->
>> <property name="messageType" value="text/plain" scope="axis2"/>
>> <property name="OUT_ONLY" value="true"/>
>> <property name="TRANSPORT_HEADERS" scope="axis2" action="remove"/>
>> <log level="full"/>
>>
>> <!-- Split any multiple syslog messages for the iterator -->
>> <!-- and return as XML with child messages -->
>> <class name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> <log level="full"/>
>> </class>
>>
>> <!-- Iterate over any multiple messages -->
>> <iterate id="syslogInterator" preservePayload="false" sequential="true" expression="//msg">
>> <log level="full"/>
>> <target>
>> <sequence>
>> <send>
>> <endpoint>
>> <address uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
>> </endpoint>
>> </send>
>> </sequence>
>> </target>
>> </iterate>
>> </inSequence>
>> <outSequence/>
>> <faultSequence>
>> <log level="full" category="ERROR" separator=","/>
>> </faultSequence>
>> </target>
>> <parameter name="transport.tcp.port">6060</parameter>
>> <parameter name="transport.tcp.contentType">text/plain</parameter>
>> </proxy>
>>
>> </definitions>
>>
>>
>>
>> -----Original Message-----
>> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> Sent: Freitag, 26. Juli 2013 20:05
>> To: user@synapse.apache.org
>> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>>
>> You should probably apply some kind of a content transformation to transform this payload into a XML payload with multiple "text" elements (one per each syslog message). But I'm not sure whether traditional techniques like XSLT and XQuery support that kind of transformations. After all you're trying to transform based on XML text and not the XML nodes. That's almost always very difficult.
>>
>> IMO you're better off writing a custom mediator to do this transformation. Then you can use the iterate mediator to split the message into its constituents and send to the JMS queue. An even more elegant solution would be to write a custom message builder that knows how to parse the incoming batch requests and construct a proper XML payload out of it.
>>
>> Thanks,
>> Hiranya
>>
>> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>>
>>> Hi,
>>>
>>> I got synapse working with a TCP-Listener accepting plain text messages
>>> as input, and the messages being sent as JMS messages to IBM Websphere MQ.
>>>
>>> Now we're in the test phase with "real" syslog (RFC-5424) data, and I'm having a problem
>>> because the TCP listener is receiving multiple syslog messages in one TCP message.
>>> The multiple messages are being sent on as one JMS message. The individual
>>> syslog messages are separated by NL and have the message length at the start.
>>>
>>> Here an example message from my log file with 8 syslog messages:
>>>
>>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">
>>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" BC="462" BK="842" Occured="1374760680" Reported="1374761340" Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"]
>>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>>> </text></soapenv:Body></soapenv:Envelope>
>>>
>>> What is the recommended way of handling this? Use the Iterate-Mediator to
>>> split the messages and send them individually? Do I have to use Xpath here,
>>> or is it possible to use e.g. javascript? Any examples would be appreciated.
>>>
>>> Thanks for any help,
>>> Peter
>>>
>>>
>>> The content of this e-mail is intended only for the confidential use of the person addressed.
>>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>>> Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>> The content of this e-mail is intended only for the confidential use of the person addressed.
>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>> Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this email immediately.
> Thank you.
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
RE: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
Posted by "Wright, Peter" <Pe...@six-group.com>.
Hi Hiranja,
Great, thanks a lot, now appears to be working correctly!!
I changed my class mediator to generate the XML as you suggested, and then had to
include the namespace in the iterate definition and include it in the xpath expression
before it worked correctly.
Best regards,
Peter
=================================
1. XML message returned by my class mediator
<?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg number 3</text></soapenv:Body></soapenv:Envelope>
2. synapse.xml fragment (iterator definition)
<iterate id="syslogInterator" preservePayload="false" sequential="true" xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
<target>
<sequence>
<send>
<endpoint>
<address .../>
</endpoint>
</send>
</sequence>
</target>
</iterate>
-----Original Message-----
From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
Sent: Freitag, 9. August 2013 01:54
To: user@synapse.apache.org
Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
> Hi Hiranja,
>
> I have now implemented a class mediator that parses the (multiple)
> syslog plain text messages and makes an XML payload from them.
> I've also got an iterate mediator that splits the messages and
> sends them individually as JMS messages to the messaging provider.
> So far so good...
>
> However, the individual JMS messages are now in XML form rather
> than plain text (with all the XML headers that I don't want).
> How can I get just the plain text payload to be sent?
>
> Thanks
> Peter
>
> =====================
>
> 1. Input: Dummy (syslog) message sent to TCP listener (individual messages separated by NL)
> 20 This is msg number 1
> 20 This is msg number 2
> 20 This is msg number 3
>
> 2. SOAP message logged by synapse
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">20 This is msg number 1
> 20 This is msg number 2
> 20 This is msg number 3
> </text></soapenv:Body></soapenv:Envelope>
>
> 3. XML message returned by my class mediator
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
This should create a message with multiple <text> elements. See the output from step 2 above. It has a single <text> element with all the content. You should break it down to multiple <text> elements. The plain text serializer of Axis2 looks for this specific <text> element when sending plain text messages.
Pay attention to the namespace (http://ws.apache.org/commons/ns/payload) of the <text> element too. It's also required.
Thanks,
Hiranya
>
> 4. JMS messages sent to the messaging provider
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 1</msg></soapenv:Body></soapenv:Envelope>
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 2</msg></soapenv:Body></soapenv:Envelope>
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 3</msg></soapenv:Body></soapenv:Envelope>
>
> 5. JMS messages that I want to be sent
> This is msg number 1
> This is msg number 2
> This is msg number 3
>
> 6. synapse.xml
> <definitions xmlns="http://ws.apache.org/ns/synapse">
> <proxy name="proxyTcp2Jms" transports="tcp">
> <target>
> <inSequence>
> <!-- Define TCP listener expects plain text (syslog) messages -->
> <property name="messageType" value="text/plain" scope="axis2"/>
> <property name="OUT_ONLY" value="true"/>
> <property name="TRANSPORT_HEADERS" scope="axis2" action="remove"/>
> <log level="full"/>
>
> <!-- Split any multiple syslog messages for the iterator -->
> <!-- and return as XML with child messages -->
> <class name="com.sixtelekurs.classMediators.syslogMsgBuilder">
> <log level="full"/>
> </class>
>
> <!-- Iterate over any multiple messages -->
> <iterate id="syslogInterator" preservePayload="false" sequential="true" expression="//msg">
> <log level="full"/>
> <target>
> <sequence>
> <send>
> <endpoint>
> <address uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
> </endpoint>
> </send>
> </sequence>
> </target>
> </iterate>
> </inSequence>
> <outSequence/>
> <faultSequence>
> <log level="full" category="ERROR" separator=","/>
> </faultSequence>
> </target>
> <parameter name="transport.tcp.port">6060</parameter>
> <parameter name="transport.tcp.contentType">text/plain</parameter>
> </proxy>
>
> </definitions>
>
>
>
> -----Original Message-----
> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> Sent: Freitag, 26. Juli 2013 20:05
> To: user@synapse.apache.org
> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>
> You should probably apply some kind of a content transformation to transform this payload into a XML payload with multiple "text" elements (one per each syslog message). But I'm not sure whether traditional techniques like XSLT and XQuery support that kind of transformations. After all you're trying to transform based on XML text and not the XML nodes. That's almost always very difficult.
>
> IMO you're better off writing a custom mediator to do this transformation. Then you can use the iterate mediator to split the message into its constituents and send to the JMS queue. An even more elegant solution would be to write a custom message builder that knows how to parse the incoming batch requests and construct a proper XML payload out of it.
>
> Thanks,
> Hiranya
>
> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>
>> Hi,
>>
>> I got synapse working with a TCP-Listener accepting plain text messages
>> as input, and the messages being sent as JMS messages to IBM Websphere MQ.
>>
>> Now we're in the test phase with "real" syslog (RFC-5424) data, and I'm having a problem
>> because the TCP listener is receiving multiple syslog messages in one TCP message.
>> The multiple messages are being sent on as one JMS message. The individual
>> syslog messages are separated by NL and have the message length at the start.
>>
>> Here an example message from my log file with 8 syslog messages:
>>
>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">
>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" BC="462" BK="842" Occured="1374760680" Reported="1374761340" Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"]
>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> </text></soapenv:Body></soapenv:Envelope>
>>
>> What is the recommended way of handling this? Use the Iterate-Mediator to
>> split the messages and send them individually? Do I have to use Xpath here,
>> or is it possible to use e.g. javascript? Any examples would be appreciated.
>>
>> Thanks for any help,
>> Peter
>>
>>
>> The content of this e-mail is intended only for the confidential use of the person addressed.
>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>> Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this email immediately.
> Thank you.
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com
The content of this e-mail is intended only for the confidential use of the person addressed.
If you are not the intended recipient, please notify the sender and delete this email immediately.
Thank you.
Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
Posted by Hiranya Jayathilaka <hi...@gmail.com>.
On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
> Hi Hiranja,
>
> I have now implemented a class mediator that parses the (multiple)
> syslog plain text messages and makes an XML payload from them.
> I've also got an iterate mediator that splits the messages and
> sends them individually as JMS messages to the messaging provider.
> So far so good...
>
> However, the individual JMS messages are now in XML form rather
> than plain text (with all the XML headers that I don't want).
> How can I get just the plain text payload to be sent?
>
> Thanks
> Peter
>
> =====================
>
> 1. Input: Dummy (syslog) message sent to TCP listener (individual messages separated by NL)
> 20 This is msg number 1
> 20 This is msg number 2
> 20 This is msg number 3
>
> 2. SOAP message logged by synapse
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">20 This is msg number 1
> 20 This is msg number 2
> 20 This is msg number 3
> </text></soapenv:Body></soapenv:Envelope>
>
> 3. XML message returned by my class mediator
> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
This should create a message with multiple <text> elements. See the output from step 2 above. It has a single <text> element with all the content. You should break it down to multiple <text> elements. The plain text serializer of Axis2 looks for this specific <text> element when sending plain text messages.
Pay attention to the namespace (http://ws.apache.org/commons/ns/payload) of the <text> element too. It's also required.
Thanks,
Hiranya
>
> 4. JMS messages sent to the messaging provider
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 1</msg></soapenv:Body></soapenv:Envelope>
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 2</msg></soapenv:Body></soapenv:Envelope>
> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg number 3</msg></soapenv:Body></soapenv:Envelope>
>
> 5. JMS messages that I want to be sent
> This is msg number 1
> This is msg number 2
> This is msg number 3
>
> 6. synapse.xml
> <definitions xmlns="http://ws.apache.org/ns/synapse">
> <proxy name="proxyTcp2Jms" transports="tcp">
> <target>
> <inSequence>
> <!-- Define TCP listener expects plain text (syslog) messages -->
> <property name="messageType" value="text/plain" scope="axis2"/>
> <property name="OUT_ONLY" value="true"/>
> <property name="TRANSPORT_HEADERS" scope="axis2" action="remove"/>
> <log level="full"/>
>
> <!-- Split any multiple syslog messages for the iterator -->
> <!-- and return as XML with child messages -->
> <class name="com.sixtelekurs.classMediators.syslogMsgBuilder">
> <log level="full"/>
> </class>
>
> <!-- Iterate over any multiple messages -->
> <iterate id="syslogInterator" preservePayload="false" sequential="true" expression="//msg">
> <log level="full"/>
> <target>
> <sequence>
> <send>
> <endpoint>
> <address uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
> </endpoint>
> </send>
> </sequence>
> </target>
> </iterate>
> </inSequence>
> <outSequence/>
> <faultSequence>
> <log level="full" category="ERROR" separator=","/>
> </faultSequence>
> </target>
> <parameter name="transport.tcp.port">6060</parameter>
> <parameter name="transport.tcp.contentType">text/plain</parameter>
> </proxy>
>
> </definitions>
>
>
>
> -----Original Message-----
> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
> Sent: Freitag, 26. Juli 2013 20:05
> To: user@synapse.apache.org
> Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages in one TCP message?
>
> You should probably apply some kind of a content transformation to transform this payload into a XML payload with multiple "text" elements (one per each syslog message). But I'm not sure whether traditional techniques like XSLT and XQuery support that kind of transformations. After all you're trying to transform based on XML text and not the XML nodes. That's almost always very difficult.
>
> IMO you're better off writing a custom mediator to do this transformation. Then you can use the iterate mediator to split the message into its constituents and send to the JMS queue. An even more elegant solution would be to write a custom message builder that knows how to parse the incoming batch requests and construct a proper XML payload out of it.
>
> Thanks,
> Hiranya
>
> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <Pe...@six-group.com> wrote:
>
>> Hi,
>>
>> I got synapse working with a TCP-Listener accepting plain text messages
>> as input, and the messages being sent as JMS messages to IBM Websphere MQ.
>>
>> Now we're in the test phase with "real" syslog (RFC-5424) data, and I'm having a problem
>> because the TCP listener is receiving multiple syslog messages in one TCP message.
>> The multiple messages are being sent on as one JMS message. The individual
>> syslog messages are separated by NL and have the message length at the start.
>>
>> Here an example message from my log file with 8 syslog messages:
>>
>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">
>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New" BC="462" BK="842" Occured="1374760680" Reported="1374761340" Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)" Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured: 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 - [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462" BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)" Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"]
>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 - [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat" timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> </text></soapenv:Body></soapenv:Envelope>
>>
>> What is the recommended way of handling this? Use the Iterate-Mediator to
>> split the messages and send them individually? Do I have to use Xpath here,
>> or is it possible to use e.g. javascript? Any examples would be appreciated.
>>
>> Thanks for any help,
>> Peter
>>
>>
>> The content of this e-mail is intended only for the confidential use of the person addressed.
>> If you are not the intended recipient, please notify the sender and delete this email immediately.
>> Thank you.
>
> --
> Hiranya Jayathilaka
> Mayhem Lab/RACE Lab;
> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
> Blog: http://techfeast-hiranya.blogspot.com
>
> The content of this e-mail is intended only for the confidential use of the person addressed.
> If you are not the intended recipient, please notify the sender and delete this email immediately.
> Thank you.
--
Hiranya Jayathilaka
Mayhem Lab/RACE Lab;
Dept. of Computer Science, UCSB; http://cs.ucsb.edu
E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
Blog: http://techfeast-hiranya.blogspot.com