You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/10/18 09:21:51 UTC
[GitHub] [pulsar-dotpulsar] amareshmad opened a new issue, #116: Dot pulsar consumer message parsing
amareshmad opened a new issue, #116:
URL: https://github.com/apache/pulsar-dotpulsar/issues/116
Hi team,
Migrating from Active MQ to dot pulsar.
Producer is java app & consumer is dotnet and after consuming messages dotnet app process some job will become a producer and java is consumer.
Active MQ: The existing behavior, java is producing the messages, dotnet consumer receives, convert back to ITextMessage , ITexMessage , nmsMessage string has nmsMessage.Properties.GetString() where we can pass property name and get back the value.
Pulsar MQ: Java is also migrated to pulsar, what they are saying messages they are sending similar to previous way (active mq),
when I take look after receiving the messages from dot net the propertied of IMessage is not having data, we have data from message.Data.ToArray().
if I extract data from
string stringData = Encoding.UTF8.GetString(message.Data.ToArray());
getting message with lot of ascii : Encoding.UTF8.GetString(message.Data.ToArray()) "��\0\u0005sr\0Bcom.Name.platform.messaging.pulsar.jms.message.PulsarTextMessage7[�{g\u001c��\u0002\0\u0001L\0\apayloadt\0\u0012Ljava/lang/String;xr\0>com.Name.platform.messaging.pulsar.jms.message.PulsarMessage�(�M.c3}\u0002\0\u0005Z\0\freadOnlyBodyL\0\acontentt\0=Lcom/Name/platform/messaging/pulsar/jms/utils/ByteSequence;L\0\aheaderst\0\u000fLjava/util/Map;L\0\tsourceMsgt\0&Lorg/apache/pulsar/client/api/Message;L\0\u0006userIDq\0~\0\u0001xp\0psr\0\u0011java.util.HashMap\u0005\a���\u0016`�\u0003\0\u0002F\0\nloadFactorI\0\tthresholdxp?@\0\0\0\0\0\u0018w\b\0\0\0 \0\0\0\u000et\0\aJMSTypet\0\vTextMessaget\0\fJMSMessageIDpt\0\u000fJMSBrokerInTimept\0\rJMSExpirationsr\0\u000ejava.lang.Long;��̏#�\u0002\0\u0001J\0\u0005valuexr\0\u0010java.lang.Number���\u001d\v���\u0002\0\0xp\0\0\0\0\0\0\0\0t\0\u000eJMSRedeliveredsr\0\u0011java.lang.Boolean� r�՜��\u0002\
0\u0001Z\0\u0005valuexp\0t\0\u0010JMSBrokerOutTimept\0\fJMSTimestampsq\0~\0\u000e\0\0\u0001�铀�t\0\u000fJMSDeliveryModesr\0\u0011java.lang.Integer\u0012⠤���8\u0002\0\u0001I\0\u0005valuexq\0~\0\u000f\0\0\0\u0002t\0\vJMSPrioritysq\0~\0\u0018\0\0\0\u0004t\0\nJMSReplyTopt\0\u000eJMSDestinationsr\0;com.Name.platform.messaging.pulsar.jms.client.PulsarQueue�U�{D���\u0002\0\u0004L\0\aoptionsq\0~\0\u0004L\0\tqueueNameq\0~\0\u0001L\0\u0019totalReceivedMessageCountt\0(Ljava/util/concurrent/atomic/AtomicLong;L\0\u0015totalSentMessageCountq\0~\0\u001fxpsq\0~\0\a?@\0\0\0\0\0\0w\b\0\0\0\u0010\0\0\0\0xt\0.Name.global.qa.NameRequestV1.Queuesr\0&java.util.concurrent.atomic.AtomicLong\u001a���w\0\u0017\u0018\u0002\0\u0001J\0\u0005valuexq\0~\0\u000f\0\0\0\0\0\0\0\0sq\0~\0#\0\0\0\0\0\0\0\0t\0\u0016JMSRegionalDestinationpt\0\u0010JMSCorrelationIDpt\0\npropertiessq\0~\0\a?@\0\0\0\0\00w\b\0\0\0@\0\0\0\u001et\0\u0016Name_des_instance_idt\0)des-nextgen-appd-rollout-7647566679-ntx
hht\0\u0013Name_des_mimetypet\0\u000fapplication/pdft\0 Name_des_proforma_reft\0mhttps://financial-e2e.platform.Name.com/v2/documents/5ffad96a-e971-4642-9f6d-5f7378527e16/sources/1t\0\u0019Name_des_document_storet\0\aFDP-DAPt\0\fName_appidt\0\u0015Name.platform.desqet\0\u0016Name_des_api_versiont\0\u0002V3t\0\u0015Name_des_start_timet\0\r1666071101578t\0\u0018Name_des_dispatch_timet\0\r1666071101578t\0\u0012Name_des_job_reft\0mhttps://financial-e2e.platform.Name.com/v2/documents/a08bd493-68c7-4684-b1dd-3533dc146aad/sources/1t\0\u000eName_des_envt\0\u0003qalt\0\vName_testt\0\u0004truet\0\u0016Name_des_polling_uriq\0~\0;t\0\u0016Name_resourceOwnerIdt\0\u000f123148206455829t\0\u0011Name_offeringidt\0\tpcglacertt\0\nName_tidt\0\u001etaximport_TY12_W2-1_1666071080t\0\u0013Name_des_job_typet\0\tTAXIMPORTt\0\u0013Name_des_is_asynct\0\u0004truet\0\u0011Name_des_job_idt\0$bb7b2b59-09a0-48da-9220-3e4d653af6abt\0\"Name_response_notification_queuet\00Name.global.qa.des.NotificationTestingV1.Que
uet\0\rName_localet\0\u0005en_USt\0\bpriorityt\0\u0006NORMALt\0\u0018Name_resourceOwnerTypet\0\u0004userq\0~\0\u001ct\0/Name.global.qa.NameResponseV1.Queuet\0\u0011singularityheadert\u0001\u000eappId=9273*ctrlguid=1665122560*acctguid=3b662425-b849-4961-afd2-c0d77ddcf9d0*ts=1666071080858*btid=1210089*snapenable=true*guid=1b7baa82-1dca-456f-b5e6-4c8726db616b*exitguid=8*unresolvedexitid=761105*cidfrom=68547*etypeorder=JMS*esubtype=JMS*cidto={[UNRESOLVED][761105]}t\0\u001bName_response_file_prefixq\0~\0Lt\0\u0017Name_des_document_refq\0~\0;t\0\u000eName_countryt\0\u0002USt\0\u0017producerMessageSentTimesr\0\rjava.time.Ser�]��\u001b\"H�\f\0\0xpw\r\u0002\0\0\0\0cN:=\"w��xt\0\u0015Name_correlation_idt\0\u001etaximport_TY12_W2-1_-519158378t\0\u0013Name_request_patht\0\u0016/v3/document/taximportxxppq\0~\0F" string
Is there any way where I can format the java format of map within C#.
One way I thought I can ask producer to change the format to JSON etc., other than this, is there any way I can convert the same message and reuse by active mq business logic (getting data based on each property) and without changing much can I use similar to active mq.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] amareshmad closed issue #116: Dot pulsar consumer message parsing
Posted by GitBox <gi...@apache.org>.
amareshmad closed issue #116: Dot pulsar consumer message parsing
URL: https://github.com/apache/pulsar-dotpulsar/issues/116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] amareshmad commented on issue #116: Dot pulsar consumer message parsing
Posted by GitBox <gi...@apache.org>.
amareshmad commented on issue #116:
URL: https://github.com/apache/pulsar-dotpulsar/issues/116#issuecomment-1283640007
@blankensteiner: pulsar supports with raw bytes & specific schema to transmit/receive.
Creating consumer with schema like string, bytearray,sequence . when I convert the consumer message data bytes to string,
stringData = Encoding.UTF8.GetString(message.Data.ToArray()). there is no change in the received stringData format.
What is best way to parse this kind of data. Producer side (java) saying they are creating properties and putting them into map, converting them and sending to pulsar server, the server is also nothing changes, when we receive lot of ascii, how can we avoid these by converting raw bytes.
I saw few people are using in Java they have PulsarTextMessage to convert the bytes streams into data. (Similar to active mq ITextMessage)
ByteArrayInputStream bais = new ByteArrayInputStream((byte[]) msg.getBody());
ObjectInputStream in = null;
PulsarTextMessage pulsarTextMessage = null;
Map<String, Object> headersMap = new HashMap<>();
try {
in = new ObjectInputStream(bais);
pulsarTextMessage = ((PulsarTextMessage) in.readObject());
System.out.println("from pulsar-jms-client:, msg is retreived from pulsarTextMessage "+pulsarTextMessage.getText());
System.out.println("from pulsar-jms-client:, headers will be retrieved from pulsarTextMessage");
System.out.println(pulsarTextMessage.getJMSDeliveryMode());
System.out.println(pulsarTextMessage.getJMSDestination());
System.out.println(pulsarTextMessage.getJMSType());
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] blankensteiner commented on issue #116: Dot pulsar consumer message parsing
Posted by GitBox <gi...@apache.org>.
blankensteiner commented on issue #116:
URL: https://github.com/apache/pulsar-dotpulsar/issues/116#issuecomment-1283583738
Pulsar just transmits the bytes. It is up to you to decide what to do with it.
You use a string payload (see DotPulsar.Schemas.StringSchema) for transmitting XML, JSON, or just a plain string.
You can also just the raw bytes (as byte[] or ReadOnlySequence\<byte>) or some of the primitive types defined in DotPulsar.Schemas.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-dotpulsar] amareshmad commented on issue #116: Dot pulsar consumer message parsing
Posted by GitBox <gi...@apache.org>.
amareshmad commented on issue #116:
URL: https://github.com/apache/pulsar-dotpulsar/issues/116#issuecomment-1291569345
this we get resolved with help of messaging team, rather than sending us the bytes data they started sending us the raw data.
that helps us to avoid parse/extract data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org