You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@synapse.apache.org by Andreas Veithen <an...@gmail.com> on 2013/09/16 18:00:15 UTC
Re: Synapse: tcp-listener: handling of multiple syslog messages in
one TCP message?
On Fri, Aug 9, 2013 at 9:26 PM, Rajika Kumarasiri
<ra...@gmail.com> wrote:
> Yeah, actually I have started working on a new syslog consumer
> implementation. As part of this I am also planning to write brand new TCP
> and UDP transports(for high volume syslog message deployments). I will
> report any progress as I go forward.
Why would the current UDP transport not be suitable for high volume messages?
Andreas
> Rajika
>
>
> On Fri, Aug 9, 2013 at 3:15 PM, Hiranya Jayathilaka <hi...@gmail.com>wrote:
>
>>
>> On Aug 9, 2013, at 4:57 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>>
>> > Hi Hiranja,
>> >
>> > Great, thanks a lot, now appears to be working correctly!!
>>
>> That's great.
>>
>> I think it would be good to have some syslog mediation support built into
>> Synapse. The correct solution is probably to implement a syslog message
>> builder and a formatter. There's a long standing jura issue regarding this
>> as well. We'll consider this feature for a future release.
>>
>> Thanks,
>> Hiranya
>>
>> > I changed my class mediator to generate the XML as you suggested, and
>> then had to
>> > include the namespace in the iterate definition and include it in the
>> xpath expression
>> > before it worked correctly.
>> >
>> > Best regards,
>> > Peter
>> >
>> > =================================
>> >
>> > 1. XML message returned by my class mediator
>> >
>> > <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">This is msg number 1</text><text
>> xmlns="http://ws.apache.org/commons/ns/payload">This is msg number
>> 2</text><text xmlns="http://ws.apache.org/commons/ns/payload">This is msg
>> number 3</text></soapenv:Body></soapenv:Envelope>
>> >
>> > 2. synapse.xml fragment (iterator definition)
>> >
>> > <iterate id="syslogInterator" preservePayload="false" sequential="true"
>> xmlns:m0="http://ws.apache.org/commons/ns/payload" expression="//m0:text">
>> > <target>
>> > <sequence>
>> > <send>
>> > <endpoint>
>> > <address .../>
>> > </endpoint>
>> > </send>
>> > </sequence>
>> > </target>
>> > </iterate>
>> >
>> > -----Original Message-----
>> > From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> > Sent: Freitag, 9. August 2013 01:54
>> > To: user@synapse.apache.org
>> > Subject: Re: Synapse: tcp-listener: handling of multiple syslog messages
>> in one TCP message?
>> >
>> >
>> > On Aug 7, 2013, at 4:53 AM, "Wright, Peter" <Pe...@six-group.com>
>> wrote:
>> >
>> >> Hi Hiranja,
>> >>
>> >> I have now implemented a class mediator that parses the (multiple)
>> >> syslog plain text messages and makes an XML payload from them.
>> >> I've also got an iterate mediator that splits the messages and
>> >> sends them individually as JMS messages to the messaging provider.
>> >> So far so good...
>> >>
>> >> However, the individual JMS messages are now in XML form rather
>> >> than plain text (with all the XML headers that I don't want).
>> >> How can I get just the plain text payload to be sent?
>> >>
>> >> Thanks
>> >> Peter
>> >>
>> >> =====================
>> >>
>> >> 1. Input: Dummy (syslog) message sent to TCP listener (individual
>> messages separated by NL)
>> >> 20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >>
>> >> 2. SOAP message logged by synapse
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">20 This is msg number 1
>> >> 20 This is msg number 2
>> >> 20 This is msg number 3
>> >> </text></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 3. XML message returned by my class mediator
>> >> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><syslogMsgs><msg>This
>> is msg number 1</msg><msg>This is msg number 2</msg><msg>This is msg number
>> 3</msg></syslogMsgs></soapenv:Body></soapenv:Envelope>
>> >
>> > This should create a message with multiple <text> elements. See the
>> output from step 2 above. It has a single <text> element with all the
>> content. You should break it down to multiple <text> elements. The plain
>> text serializer of Axis2 looks for this specific <text> element when
>> sending plain text messages.
>> >
>> > Pay attention to the namespace (http://ws.apache.org/commons/ns/payload)
>> of the <text> element too. It's also required.
>> >
>> > Thanks,
>> > Hiranya
>> >
>> >>
>> >> 4. JMS messages sent to the messaging provider
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 1</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 2</msg></soapenv:Body></soapenv:Envelope>
>> >> <?xml version='1.0' encoding='UTF-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><msg>This is msg
>> number 3</msg></soapenv:Body></soapenv:Envelope>
>> >>
>> >> 5. JMS messages that I want to be sent
>> >> This is msg number 1
>> >> This is msg number 2
>> >> This is msg number 3
>> >>
>> >> 6. synapse.xml
>> >> <definitions xmlns="http://ws.apache.org/ns/synapse">
>> >> <proxy name="proxyTcp2Jms" transports="tcp">
>> >> <target>
>> >> <inSequence>
>> >> <!-- Define TCP listener expects plain text (syslog)
>> messages -->
>> >> <property name="messageType" value="text/plain"
>> scope="axis2"/>
>> >> <property name="OUT_ONLY" value="true"/>
>> >> <property name="TRANSPORT_HEADERS" scope="axis2"
>> action="remove"/>
>> >> <log level="full"/>
>> >>
>> >> <!-- Split any multiple syslog messages for the iterator
>> -->
>> >> <!-- and return as XML with child messages
>> -->
>> >> <class
>> name="com.sixtelekurs.classMediators.syslogMsgBuilder">
>> >> <log level="full"/>
>> >> </class>
>> >>
>> >> <!-- Iterate over any multiple messages -->
>> >> <iterate id="syslogInterator" preservePayload="false"
>> sequential="true" expression="//msg">
>> >> <log level="full"/>
>> >> <target>
>> >> <sequence>
>> >> <send>
>> >> <endpoint>
>> >> <address
>> uri="jms:/cn=sed.finesb.syslog?java.naming.factory.initial=com.sun.jndi.ldap.LdapCtxFactory&java.naming.provider.url=MY_LDAP_URL&transport.jms.ConnectionFactoryJNDIName=cn=MY_TCP&transport.jms.DestinationType=topic&java.naming.security.principal=MY_DN&java.naming.security.credentials=MY_PW"/>
>> >> </endpoint>
>> >> </send>
>> >> </sequence>
>> >> </target>
>> >> </iterate>
>> >> </inSequence>
>> >> <outSequence/>
>> >> <faultSequence>
>> >> <log level="full" category="ERROR" separator=","/>
>> >> </faultSequence>
>> >> </target>
>> >> <parameter name="transport.tcp.port">6060</parameter>
>> >> <parameter name="transport.tcp.contentType">text/plain</parameter>
>> >> </proxy>
>> >>
>> >> </definitions>
>> >>
>> >>
>> >>
>> >> -----Original Message-----
>> >> From: Hiranya Jayathilaka [mailto:hiranya911@gmail.com]
>> >> Sent: Freitag, 26. Juli 2013 20:05
>> >> To: user@synapse.apache.org
>> >> Subject: Re: Synapse: tcp-listener: handling of multiple syslog
>> messages in one TCP message?
>> >>
>> >> You should probably apply some kind of a content transformation to
>> transform this payload into a XML payload with multiple "text" elements
>> (one per each syslog message). But I'm not sure whether traditional
>> techniques like XSLT and XQuery support that kind of transformations. After
>> all you're trying to transform based on XML text and not the XML nodes.
>> That's almost always very difficult.
>> >>
>> >> IMO you're better off writing a custom mediator to do this
>> transformation. Then you can use the iterate mediator to split the message
>> into its constituents and send to the JMS queue. An even more elegant
>> solution would be to write a custom message builder that knows how to parse
>> the incoming batch requests and construct a proper XML payload out of it.
>> >>
>> >> Thanks,
>> >> Hiranya
>> >>
>> >> On Jul 26, 2013, at 6:10 AM, "Wright, Peter" <
>> Peter.Wright@six-group.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> I got synapse working with a TCP-Listener accepting plain text messages
>> >>> as input, and the messages being sent as JMS messages to IBM Websphere
>> MQ.
>> >>>
>> >>> Now we're in the test phase with "real" syslog (RFC-5424) data, and
>> I'm having a problem
>> >>> because the TCP listener is receiving multiple syslog messages in one
>> TCP message.
>> >>> The multiple messages are being sent on as one JMS message. The
>> individual
>> >>> syslog messages are separated by NL and have the message length at the
>> start.
>> >>>
>> >>> Here an example message from my log file with 8 syslog messages:
>> >>>
>> >>> 2013-07-25T16:12:20,INFO ,mdzhsfesb01,MSGMED,MediatorLog.java:99,To: ,
>> WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID:
>> urn:uuid:1BC21F33F4BAA449111374761227027, Direction: request, Envelope:
>> >>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="
>> http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="
>> http://ws.apache.org/commons/ns/payload">
>> >>> 186 <174>1 2013-07-25T16:08:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="1"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761220" telegrams="5522922" BKs="632" LineIDs="441"]
>> >>> 400 <170>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="2"] [Ticket_605@7410 Type="Feed Outage" Status="New"
>> BC="462" BK="842" Occured="1374760680" Reported="1374761340"
>> Summary="Outage of BC 462 at 2013-07-25 13:58:00 UTC (DAWD on mdzhldos02)"
>> Details="Outage of BC 462; BK 842; No Bid/Asks for 11 minutes; Occured:
>> 2013-07-25 13:58:00 UTC; Reported by DAWD on mdzhldos02"]
>> >>> 186 <174>1 2013-07-25T16:09:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="3"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761280" telegrams="6001019" BKs="665" LineIDs="441"]
>> >>> 186 <174>1 2013-07-25T16:10:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="4"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761340" telegrams="4908793" BKs="612" LineIDs="440"]
>> >>> 119 <174>1 2013-07-25T16:10:40+02:00 mdzhldos02 DaMon.DAWD 24438 -
>> [meta sequenceId="5"] [HEARTBEAT@7410 type="Heartbeat"]
>> >>> 362 <170>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="6"] [Ticket_605@7410 Status="Close" BC="462"
>> BK="(null)" Occured="1374761340" Reported="1374761460" Summary="Problem
>> with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD on mdzhldos02)"
>> Details="Problem with BC 462 disappeared at 2013-07-25 14:09:00 UTC (DAWD
>> on mdzhldos02)"]
>> >>> 186 <174>1 2013-07-25T16:11:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="7"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761400" telegrams="6099578" BKs="657" LineIDs="440"]
>> >>> 186 <174>1 2013-07-25T16:12:00+02:00 mdzhldos02 DaMon.DAWD 12355 -
>> [meta sequenceId="8"] [HEARTBEAT@7410 Type="Heartbeat"
>> timestamp="1374761460" telegrams="5570590" BKs="613" LineIDs="446"]
>> >>> </text></soapenv:Body></soapenv:Envelope>
>> >>>
>> >>> What is the recommended way of handling this? Use the Iterate-Mediator
>> to
>> >>> split the messages and send them individually? Do I have to use Xpath
>> here,
>> >>> or is it possible to use e.g. javascript? Any examples would be
>> appreciated.
>> >>>
>> >>> Thanks for any help,
>> >>> Peter
>> >>>
>> >>>
>> >>> The content of this e-mail is intended only for the confidential use
>> of the person addressed.
>> >>> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >>> Thank you.
>> >>
>> >> --
>> >> Hiranya Jayathilaka
>> >> Mayhem Lab/RACE Lab;
>> >> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> >> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> >> Blog: http://techfeast-hiranya.blogspot.com
>> >>
>> >> The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> >> If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> >> Thank you.
>> >
>> > --
>> > Hiranya Jayathilaka
>> > Mayhem Lab/RACE Lab;
>> > Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> > E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> > Blog: http://techfeast-hiranya.blogspot.com
>> >
>> > The content of this e-mail is intended only for the confidential use of
>> the person addressed.
>> > If you are not the intended recipient, please notify the sender and
>> delete this email immediately.
>> > Thank you.
>>
>> --
>> Hiranya Jayathilaka
>> Mayhem Lab/RACE Lab;
>> Dept. of Computer Science, UCSB; http://cs.ucsb.edu
>> E-mail: hiranya@cs.ucsb.edu; Mobile: +1 (805) 895-7443
>> Blog: http://techfeast-hiranya.blogspot.com
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@synapse.apache.org
For additional commands, e-mail: dev-help@synapse.apache.org