TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

Hello,

We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.

We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.

With this option checked:

a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.  
Conversely it reads a byte array and converts it to a hexadecimal string.

b) The TCPClient has been extended to support the following new methods:
void write(OutputStream os, byte[] b);
byte [] readLengthPrefixed(InputStream is); 
The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.

e.g. 
Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020" 
Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]

This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal. 

There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?

Thanks,
Oghie.   





 
**********************************************************************
 
E-mail disclaimer
FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
 
This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
 
**********************************************************************

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

> Yes, I can do the updates to the TCPSampler.

> I think you can still test your code using the current JMeter, so long
> as you use the old TCPClient interface methods. Converting to the new
> interface could be done later when the TCP Sampler is updated. Should
> be fairly trivial - I'm happy to do that.

Thanks, will use existing interface.

-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 10 October 2008 16:27
To: JMeter Developers List
Cc: Mike Cronin
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > Yes, but the character set is largely irrelevant, as the hex
>  > characters are the same in most (all?) sets.
>
>
> > I was assuming that the hex-encoded binary string would be converted
>  > to binary in the client. This does not really need a variable encoding
>  > as hex characters are the same in all main character sets, but for
>  > portability the client could use a fixed set, e.g. 8859-1 or utf-8.
>  >
>  > Likewise for the binary->hex conversion. The client could convert
>  > binary nibbles to character string and then to bytes, but since we
>  > know the conversion for these they could be converted direct to bytes.
>  >
>  > The character set used to display the response would not matter in this case.
>
>
> Thanks, needed to be sure we were talking about the same thing and wondered whether hex representation was different according to character set.
>
>  So guess I'm back to what we agreed a few days ago...
>
>  Will create client(s) to support hex-binary encoding and length prefix (length of prefix specified in properties for now).  Will create a local version of the TCPClient Interface which you outlined earlier and implement this interface.
>

OK.

>  Locally the client will be selectable from the TCP Sampler GUI as you originally suggested.  We need this flexibility across TCP test plans where the same user would wish to run the TCPClientImpl in some and the new client in another without having to change the properties file in-between.

Yes, I can do the updates to the TCPSampler.

I think you can still test your code using the current JMeter, so long
as you use the old TCPClient interface methods. Converting to the new
interface could be done later when the TCP Sampler is updated. Should
be fairly trivial - I'm happy to do that.

>  Can retrofit later if you think we should have a separate TCPSampler/GUI.
>
>
>  > I suppose the question is: are there any protocols where it would be
>  > difficult/wasteful to return bytes rather than a String?
>
>  > If so, then perhaps the SampleResult class needs to be enhanced to
>  > handle results returned as Strings as well as byte arrays. I'm not
>  > sure how easy that would be.
>
>  > There would also need to be a way to tell the TCP sampler that the
>  > client returns a String rather than a byte array - assuming we change
>  > the read() method.
>
>  > I need to think some more about this !
>
>
> Could not come up with a realistic example.
>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 10 October 2008 11:37
>  To: JMeter Developers List
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > > WDYT?
>  >
>  >  Overall I think this is a good idea because, if my understanding is correct, it allows the sampler->client->sampler->listener to use distinct character encodings.  Also it puts the encoding logic in the endpoints where it is needed rather than cluttering up the sampler.
>  >
>  >  I am a little confused in relation to the encoding/decoding of the hex-encoded binary because there is a double-encoding here as the string containing the hex-encoded binary is itself encoded in a character set.
>
>  Yes, but the character set is largely irrelevant, as the hex
>  characters are the same in most (all?) sets.
>
>  >  byte array->hex-encoded binary string->byte array->character set encoded string
>  >
>  >  On receiving a response the byte array would need to be converted to a hex-encoded binary string (in the default character set) then back to a byte array (using String.getBytes()) then finally into the target character set.
>  >
>  >  I think that you meant the byte array->hex-encoded binary string->byte array conversion to occur within the client and the character set encoding to occur within the listener, is this correct?
>  >
>  >
>
>  I was assuming that the hex-encoded binary string would be converted
>  to binary in the client. This does not really need a variable encoding
>  as hex characters are the same in all main character sets, but for
>  portability the client could use a fixed set, e.g. 8859-1 or utf-8.
>
>  Likewise for the binary->hex conversion. The client could convert
>  binary nibbles to character string and then to bytes, but since we
>  know the conversion for these they could be converted direct to bytes.
>
>  The character set used to display the response would not matter in this case.
>
>  I suppose the question is: are there any protocols where it would be
>  difficult/wasteful to return bytes rather than a String?
>
>  If so, then perhaps the SampleResult class needs to be enhanced to
>  handle results returned as Strings as well as byte arrays. I'm not
>  sure how easy that would be.
>
>  There would also need to be a way to tell the TCP sampler that the
>  client returns a String rather than a byte array - assuming we change
>  the read() method.
>
>  I need to think some more about this !
>
>  >
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >
>  > Sent: 09 October 2008 13:34
>  >  To: JMeter Developers List
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > > Yes.
>  >  >
>  >  >  > I'm still considering how to fix up the read() method. Probably I'll
>  >  >  > add a readBytes() method and maybe add an abstract class that
>  >  >  > implements it.
>  >  >
>  >  >  > The problem is that the sampler has to convert the string to bytes in
>  >  >  > order to store it in the sample result. Wire protocols work in terms
>  >  >  > of bytes, so this means multiple conversions, with the possibility of
>  >  >  > data loss if some bytes are not both-ways convertible.
>  >  >
>  >  >
>  >  > Am with you now, had thought that strings were stored in the sample result but of course they're not.
>  >  >
>  >  >  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>  >  >
>  >  >  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?
>  >
>  >  Yes, that's possible. In either case existing implementations would
>  >  need to be amended and rebuilt, so I suppose it does not matter which
>  >  is done.
>  >
>  >  It would be possible to extend the TCPClient interface to add a
>  >  readBytes() method, and have TCPSampler use instanceof to choose which
>  >  method to use. This would allow old implementations to continue, but
>  >  would be less efficient - and we want implementations to return bytes.
>  >  So I think it would be OK to break compatibility.
>  >
>  >  >  Also think that a writeBytes() method is needed.
>  >  >
>  >
>  >  I see what you mean - the client implementation currently has to
>  >  convert the string into bytes, and the sampler has no control over the
>  >  encoding that is used.
>  >
>  >  I think it's unlikely that there are many 3rd party implementations
>  >  (if any) so perhaps the string method should be removed.
>  >
>  >  We need the input/output encodings to be specified on the sampler GUI
>  >  (they can default to the current setting, for backward compat.)
>  >
>  >  >  > Sorry, should have explained in more detail: I was thinking that setup
>  >  >  > parameters could be used to configure the protocol implementation.
>  >  >  > This could be useful for minor variants on the protocol - e.g. 4byte
>  >  >  > length prefix, and variable length strings.
>  >  >
>  >  >
>  >  > Yes this would make sense.
>  >  >
>  >  >  ******
>  >  >
>  >  >  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...
>  >
>  >  Good idea.
>  >
>  >  >  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.
>  >
>  >  Yes.
>  >
>  >  >  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.
>  >
>  >  I was suggesting that the sampler deals only in the data without prefixes.
>  >  The client needs to know how to add/remove the length prefix.
>  >  The prefix length is either fixed in the client, or the sampler can
>  >  pass in a parameter which tells it what particular protocol variation
>  >  is to be used.
>  >
>  >  The sampler is interested in the user data, the client looks after the
>  >  packaging.
>  >
>  >  >  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>  >  >
>  >  >  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.
>  >
>  >  Yes.
>  >
>  >  >  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.
>  >
>  >  This will only be true if the input is bytes or a stream.
>  >  I now think the client show know the encoding (see below).
>  >
>  >  >  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.
>  >
>  >  I think that depends on how you look at the protocol. In your original
>  >  case, you would like JMeter to hande hex-encoded binary. You could
>  >  regard the conversion between hex and binary as part of the protocol,
>  >  and therefore handled by the client.
>  >
>  >  I'm not keen on putting too much processing in the sampler. The idea
>  >  is that JMeter users can develop their own implementations in order to
>  >  handle any special processing, and this can be provided in an add-on
>  >  jar. It's a lot harder for a user to make changes to the sampler.
>  >
>  >  I'm hoping that we can come up with more generic sampler which allows
>  >  client implementations to do what they need to do.
>  >
>  >  I'm now beginning to think that the client should still be passed the
>  >  string from the GUI, but that it also needs the encoding - which it
>  >  may ignore. This makes it more flexible for clients that need to
>  >  process the string before transmission.
>  >
>  >  >  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.
>  >
>  >  Yes, see above.
>  >
>  >  >  To achieve this I now think that the following changes should be made:
>  >  >
>  >  >  Client:
>  >  >  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  >  >  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>  >  >
>  >  >  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.
>  >
>  >  Yes, that's what I was thinking.
>  >
>  >  >  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.
>  >
>  >  Yes.
>  >
>  >  >  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.
>  >
>  >  Agreed.
>  >
>  >  >  Sampler GUI:
>  >  >  From above GUI needs to specify
>  >  >  a) Length prefix length
>  >  >  b) Length prefix format OR ClientImpl as above
>  >
>  >  No, I think these are part of the client implementation, see above.
>  >
>  >  >  In addition the GUI needs to specify
>  >  >  c) Data format
>  >
>  >  Again, I think this is best dealt with by the client implementation.
>  >
>  >  >  Sampler:
>  >  >  The sampler needs to convert the input text string into a byte[] for transmission and a
>  >  byte[] into a text string (e.g. to hex string for binary/to local
>  >  character set for character set).  This then goes back into the
>  >  sampler result using the getBytes() method as before.
>  >
>  >  Not quite.
>  >
>  >  Originally I thought the sampler should convert the GUI data to bytes
>  >  using the specified encoding. I now think the sampler should pass in
>  >  the string and the encoding, as this is more flexible.
>  >
>  >  On return, it copies the bytes to the sample result, and sets the
>  >  appropriate encoding - there is no conversion at that point. The
>  >  response is only converted to a string when it is extracted from the
>  >  sample result, e.g. by the Tree View Listener.
>  >
>  >  The only issue I can see here is: how does the sampler know what
>  >  encoding to use for the response? This can be specified on the GUI (or
>  >  we can assume it is the same as the request encoding). If there is
>  >  likely to be a possibility that the encoding is defined in the
>  >  response message (for example, this is how HTTP responses work), then
>  >  I think it makes sense for the client to extract this information, and
>  >  pass it back to the sampler somehow.
>  >
>  >  We could add a getEncoding() method to the interface, but it might be
>  >  better to use a generic Object getParameter(String name) to the
>  >  interface, in case it is later found that there are other items that
>  >  need to be retrieved. The sampler would need to be updated, but
>  >  existing clients would not need to be updated.
>  >
>  >  In summary, I think the TCPClient interface should probably be amended
>  >  as follows:
>  >
>  >  void write(OutputStream os, String s, String encoding);
>  >
>  >  The
>  >     void write(OutputStream os, InputStream is);
>  >  method probably does not need the encoding.
>  >
>  >  void setParameter(String name, Object value);
>  >
>  >  Object getParameter(String name);
>  >
>  >  There would be an abstract class that provides a dummy implementation
>  >  for the get/set Parameter methods.
>  >
>  >  WDYT?
>  >
>  >  >  ******
>  >  >
>  >  >
>  >  >  -----Original Message-----
>  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >
>  >  > Sent: 08 October 2008 16:57
>  >  >  To: JMeter Developers List
>  >  >  Cc: Mike Cronin
>  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  data
>  >  >
>  >  >
>  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >  >  >
>  >  >  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >  >  >
>  >  >
>  >  >  Yes.
>  >  >
>  >  >  I'm still considering how to fix up the read() method. Probably I'll
>  >  >  add a readBytes() method and maybe add an abstract class that
>  >  >  implements it.
>  >  >
>  >  >  The problem is that the sampler has to convert the string to bytes in
>  >  >  order to store it in the sample result. Wire protocols work in terms
>  >  >  of bytes, so this means multiple conversions, with the possibility of
>  >  >  data loss if some bytes are not both-ways convertible.
>  >  >
>  >  >  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >  >  >
>  >  >
>  >  >  Sorry, should have explained in more detail: I was thinking that setup
>  >  >  parameters could be used to configure the protocol implementation.
>  >  >  This could be useful for minor variants on the protocol - e.g. 4byte
>  >  >  length prefix, and variable length strings.
>  >  >
>  >  >  >
>  >  >  >  -----Original Message-----
>  >  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >  >  Sent: 08 October 2008 13:04
>  >  >  >  To: JMeter Developers List
>  >  >  >  Cc: Mike Cronin
>  >  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  >  data
>  >  >  >
>  >  >  >
>  >  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  >  > Hello,
>  >  >  >  >
>  >  >  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >  >  >
>  >  >  >  Is this a proprietary service, or are there several such services?
>  >  >  >
>  >  >  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >  >  >
>  >  >  >  >  With this option checked:
>  >  >  >  >
>  >  >  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >  >  >
>  >  >  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  >  >  void write(OutputStream os, byte[] b);
>  >  >  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >  >  >
>  >  >  >  >  e.g.
>  >  >  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >  >  >
>  >  >  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >  >  >
>  >  >  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >  >  >
>  >  >  >
>  >  >  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >  >  >
>  >  >  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  >  >  a new TCPClient implementation?
>  >  >  >
>  >  >  >  At present the TCPSampler only supports a single implementation that
>  >  >  >  is defined via a property, but it could be enhanced to fetch the name
>  >  >  >  from the GUI to support test plans which require more than one
>  >  >  >  implementation.
>  >  >  >
>  >  >  >  It might be necessary to extend the TCPClient interface to support
>  >  >  >  some kind of setup method to allow parameters to be passed in; these
>  >  >  >  would need to be added to the TCP Sample GUI.
>  >  >  >
>  >  >  >  The advantage of this approach would be that there should be no need
>  >  >  >  to change the TCP Sampler further in order to accommodate new
>  >  >  >  protocols.
>  >  >  >
>  >  >  >  I have just realised that the TCPClient interface read() method was
>  >  >  >  badly designed; it should have returned a byte array. At present both
>  >  >  >  the client and the sampler have to do String<>byte conversions - and
>  >  >  >  there is no provision for specifying the character set to be used.
>  >  >  >
>  >  >  >  However, that does not affect your current requirements.
>  >  >  >
>  >  >  >  >  Thanks,
>  >  >  >  >  Oghie.
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >  **********************************************************************
>  >  >  >  >
>  >  >  >  >  E-mail disclaimer
>  >  >  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >  >  >
>  >  >  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >  >  >
>  >  >  >  >  **********************************************************************
>  >  >  >  >
>  >  >  >  >  ---------------------------------------------------------------------
>  >  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >  >
>  >  >  >  >
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

Re: TCP Sampler Extension to support length-prefixed binary data

[sebb <se...@gmail.com>]

On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > Yes, but the character set is largely irrelevant, as the hex
>  > characters are the same in most (all?) sets.
>
>
> > I was assuming that the hex-encoded binary string would be converted
>  > to binary in the client. This does not really need a variable encoding
>  > as hex characters are the same in all main character sets, but for
>  > portability the client could use a fixed set, e.g. 8859-1 or utf-8.
>  >
>  > Likewise for the binary->hex conversion. The client could convert
>  > binary nibbles to character string and then to bytes, but since we
>  > know the conversion for these they could be converted direct to bytes.
>  >
>  > The character set used to display the response would not matter in this case.
>
>
> Thanks, needed to be sure we were talking about the same thing and wondered whether hex representation was different according to character set.
>
>  So guess I'm back to what we agreed a few days ago...
>
>  Will create client(s) to support hex-binary encoding and length prefix (length of prefix specified in properties for now).  Will create a local version of the TCPClient Interface which you outlined earlier and implement this interface.
>

OK.

>  Locally the client will be selectable from the TCP Sampler GUI as you originally suggested.  We need this flexibility across TCP test plans where the same user would wish to run the TCPClientImpl in some and the new client in another without having to change the properties file in-between.

Yes, I can do the updates to the TCPSampler.

I think you can still test your code using the current JMeter, so long
as you use the old TCPClient interface methods. Converting to the new
interface could be done later when the TCP Sampler is updated. Should
be fairly trivial - I'm happy to do that.

>  Can retrofit later if you think we should have a separate TCPSampler/GUI.
>
>
>  > I suppose the question is: are there any protocols where it would be
>  > difficult/wasteful to return bytes rather than a String?
>
>  > If so, then perhaps the SampleResult class needs to be enhanced to
>  > handle results returned as Strings as well as byte arrays. I'm not
>  > sure how easy that would be.
>
>  > There would also need to be a way to tell the TCP sampler that the
>  > client returns a String rather than a byte array - assuming we change
>  > the read() method.
>
>  > I need to think some more about this !
>
>
> Could not come up with a realistic example.
>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 10 October 2008 11:37
>  To: JMeter Developers List
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > > WDYT?
>  >
>  >  Overall I think this is a good idea because, if my understanding is correct, it allows the sampler->client->sampler->listener to use distinct character encodings.  Also it puts the encoding logic in the endpoints where it is needed rather than cluttering up the sampler.
>  >
>  >  I am a little confused in relation to the encoding/decoding of the hex-encoded binary because there is a double-encoding here as the string containing the hex-encoded binary is itself encoded in a character set.
>
>  Yes, but the character set is largely irrelevant, as the hex
>  characters are the same in most (all?) sets.
>
>  >  byte array->hex-encoded binary string->byte array->character set encoded string
>  >
>  >  On receiving a response the byte array would need to be converted to a hex-encoded binary string (in the default character set) then back to a byte array (using String.getBytes()) then finally into the target character set.
>  >
>  >  I think that you meant the byte array->hex-encoded binary string->byte array conversion to occur within the client and the character set encoding to occur within the listener, is this correct?
>  >
>  >
>
>  I was assuming that the hex-encoded binary string would be converted
>  to binary in the client. This does not really need a variable encoding
>  as hex characters are the same in all main character sets, but for
>  portability the client could use a fixed set, e.g. 8859-1 or utf-8.
>
>  Likewise for the binary->hex conversion. The client could convert
>  binary nibbles to character string and then to bytes, but since we
>  know the conversion for these they could be converted direct to bytes.
>
>  The character set used to display the response would not matter in this case.
>
>  I suppose the question is: are there any protocols where it would be
>  difficult/wasteful to return bytes rather than a String?
>
>  If so, then perhaps the SampleResult class needs to be enhanced to
>  handle results returned as Strings as well as byte arrays. I'm not
>  sure how easy that would be.
>
>  There would also need to be a way to tell the TCP sampler that the
>  client returns a String rather than a byte array - assuming we change
>  the read() method.
>
>  I need to think some more about this !
>
>  >
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >
>  > Sent: 09 October 2008 13:34
>  >  To: JMeter Developers List
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > > Yes.
>  >  >
>  >  >  > I'm still considering how to fix up the read() method. Probably I'll
>  >  >  > add a readBytes() method and maybe add an abstract class that
>  >  >  > implements it.
>  >  >
>  >  >  > The problem is that the sampler has to convert the string to bytes in
>  >  >  > order to store it in the sample result. Wire protocols work in terms
>  >  >  > of bytes, so this means multiple conversions, with the possibility of
>  >  >  > data loss if some bytes are not both-ways convertible.
>  >  >
>  >  >
>  >  > Am with you now, had thought that strings were stored in the sample result but of course they're not.
>  >  >
>  >  >  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>  >  >
>  >  >  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?
>  >
>  >  Yes, that's possible. In either case existing implementations would
>  >  need to be amended and rebuilt, so I suppose it does not matter which
>  >  is done.
>  >
>  >  It would be possible to extend the TCPClient interface to add a
>  >  readBytes() method, and have TCPSampler use instanceof to choose which
>  >  method to use. This would allow old implementations to continue, but
>  >  would be less efficient - and we want implementations to return bytes.
>  >  So I think it would be OK to break compatibility.
>  >
>  >  >  Also think that a writeBytes() method is needed.
>  >  >
>  >
>  >  I see what you mean - the client implementation currently has to
>  >  convert the string into bytes, and the sampler has no control over the
>  >  encoding that is used.
>  >
>  >  I think it's unlikely that there are many 3rd party implementations
>  >  (if any) so perhaps the string method should be removed.
>  >
>  >  We need the input/output encodings to be specified on the sampler GUI
>  >  (they can default to the current setting, for backward compat.)
>  >
>  >  >  > Sorry, should have explained in more detail: I was thinking that setup
>  >  >  > parameters could be used to configure the protocol implementation.
>  >  >  > This could be useful for minor variants on the protocol - e.g. 4byte
>  >  >  > length prefix, and variable length strings.
>  >  >
>  >  >
>  >  > Yes this would make sense.
>  >  >
>  >  >  ******
>  >  >
>  >  >  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...
>  >
>  >  Good idea.
>  >
>  >  >  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.
>  >
>  >  Yes.
>  >
>  >  >  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.
>  >
>  >  I was suggesting that the sampler deals only in the data without prefixes.
>  >  The client needs to know how to add/remove the length prefix.
>  >  The prefix length is either fixed in the client, or the sampler can
>  >  pass in a parameter which tells it what particular protocol variation
>  >  is to be used.
>  >
>  >  The sampler is interested in the user data, the client looks after the
>  >  packaging.
>  >
>  >  >  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>  >  >
>  >  >  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.
>  >
>  >  Yes.
>  >
>  >  >  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.
>  >
>  >  This will only be true if the input is bytes or a stream.
>  >  I now think the client show know the encoding (see below).
>  >
>  >  >  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.
>  >
>  >  I think that depends on how you look at the protocol. In your original
>  >  case, you would like JMeter to hande hex-encoded binary. You could
>  >  regard the conversion between hex and binary as part of the protocol,
>  >  and therefore handled by the client.
>  >
>  >  I'm not keen on putting too much processing in the sampler. The idea
>  >  is that JMeter users can develop their own implementations in order to
>  >  handle any special processing, and this can be provided in an add-on
>  >  jar. It's a lot harder for a user to make changes to the sampler.
>  >
>  >  I'm hoping that we can come up with more generic sampler which allows
>  >  client implementations to do what they need to do.
>  >
>  >  I'm now beginning to think that the client should still be passed the
>  >  string from the GUI, but that it also needs the encoding - which it
>  >  may ignore. This makes it more flexible for clients that need to
>  >  process the string before transmission.
>  >
>  >  >  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.
>  >
>  >  Yes, see above.
>  >
>  >  >  To achieve this I now think that the following changes should be made:
>  >  >
>  >  >  Client:
>  >  >  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  >  >  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>  >  >
>  >  >  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.
>  >
>  >  Yes, that's what I was thinking.
>  >
>  >  >  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.
>  >
>  >  Yes.
>  >
>  >  >  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.
>  >
>  >  Agreed.
>  >
>  >  >  Sampler GUI:
>  >  >  From above GUI needs to specify
>  >  >  a) Length prefix length
>  >  >  b) Length prefix format OR ClientImpl as above
>  >
>  >  No, I think these are part of the client implementation, see above.
>  >
>  >  >  In addition the GUI needs to specify
>  >  >  c) Data format
>  >
>  >  Again, I think this is best dealt with by the client implementation.
>  >
>  >  >  Sampler:
>  >  >  The sampler needs to convert the input text string into a byte[] for transmission and a
>  >  byte[] into a text string (e.g. to hex string for binary/to local
>  >  character set for character set).  This then goes back into the
>  >  sampler result using the getBytes() method as before.
>  >
>  >  Not quite.
>  >
>  >  Originally I thought the sampler should convert the GUI data to bytes
>  >  using the specified encoding. I now think the sampler should pass in
>  >  the string and the encoding, as this is more flexible.
>  >
>  >  On return, it copies the bytes to the sample result, and sets the
>  >  appropriate encoding - there is no conversion at that point. The
>  >  response is only converted to a string when it is extracted from the
>  >  sample result, e.g. by the Tree View Listener.
>  >
>  >  The only issue I can see here is: how does the sampler know what
>  >  encoding to use for the response? This can be specified on the GUI (or
>  >  we can assume it is the same as the request encoding). If there is
>  >  likely to be a possibility that the encoding is defined in the
>  >  response message (for example, this is how HTTP responses work), then
>  >  I think it makes sense for the client to extract this information, and
>  >  pass it back to the sampler somehow.
>  >
>  >  We could add a getEncoding() method to the interface, but it might be
>  >  better to use a generic Object getParameter(String name) to the
>  >  interface, in case it is later found that there are other items that
>  >  need to be retrieved. The sampler would need to be updated, but
>  >  existing clients would not need to be updated.
>  >
>  >  In summary, I think the TCPClient interface should probably be amended
>  >  as follows:
>  >
>  >  void write(OutputStream os, String s, String encoding);
>  >
>  >  The
>  >     void write(OutputStream os, InputStream is);
>  >  method probably does not need the encoding.
>  >
>  >  void setParameter(String name, Object value);
>  >
>  >  Object getParameter(String name);
>  >
>  >  There would be an abstract class that provides a dummy implementation
>  >  for the get/set Parameter methods.
>  >
>  >  WDYT?
>  >
>  >  >  ******
>  >  >
>  >  >
>  >  >  -----Original Message-----
>  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >
>  >  > Sent: 08 October 2008 16:57
>  >  >  To: JMeter Developers List
>  >  >  Cc: Mike Cronin
>  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  data
>  >  >
>  >  >
>  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >  >  >
>  >  >  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >  >  >
>  >  >
>  >  >  Yes.
>  >  >
>  >  >  I'm still considering how to fix up the read() method. Probably I'll
>  >  >  add a readBytes() method and maybe add an abstract class that
>  >  >  implements it.
>  >  >
>  >  >  The problem is that the sampler has to convert the string to bytes in
>  >  >  order to store it in the sample result. Wire protocols work in terms
>  >  >  of bytes, so this means multiple conversions, with the possibility of
>  >  >  data loss if some bytes are not both-ways convertible.
>  >  >
>  >  >  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >  >  >
>  >  >
>  >  >  Sorry, should have explained in more detail: I was thinking that setup
>  >  >  parameters could be used to configure the protocol implementation.
>  >  >  This could be useful for minor variants on the protocol - e.g. 4byte
>  >  >  length prefix, and variable length strings.
>  >  >
>  >  >  >
>  >  >  >  -----Original Message-----
>  >  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >  >  Sent: 08 October 2008 13:04
>  >  >  >  To: JMeter Developers List
>  >  >  >  Cc: Mike Cronin
>  >  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  >  data
>  >  >  >
>  >  >  >
>  >  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  >  > Hello,
>  >  >  >  >
>  >  >  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >  >  >
>  >  >  >  Is this a proprietary service, or are there several such services?
>  >  >  >
>  >  >  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >  >  >
>  >  >  >  >  With this option checked:
>  >  >  >  >
>  >  >  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >  >  >
>  >  >  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  >  >  void write(OutputStream os, byte[] b);
>  >  >  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >  >  >
>  >  >  >  >  e.g.
>  >  >  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >  >  >
>  >  >  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >  >  >
>  >  >  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >  >  >
>  >  >  >
>  >  >  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >  >  >
>  >  >  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  >  >  a new TCPClient implementation?
>  >  >  >
>  >  >  >  At present the TCPSampler only supports a single implementation that
>  >  >  >  is defined via a property, but it could be enhanced to fetch the name
>  >  >  >  from the GUI to support test plans which require more than one
>  >  >  >  implementation.
>  >  >  >
>  >  >  >  It might be necessary to extend the TCPClient interface to support
>  >  >  >  some kind of setup method to allow parameters to be passed in; these
>  >  >  >  would need to be added to the TCP Sample GUI.
>  >  >  >
>  >  >  >  The advantage of this approach would be that there should be no need
>  >  >  >  to change the TCP Sampler further in order to accommodate new
>  >  >  >  protocols.
>  >  >  >
>  >  >  >  I have just realised that the TCPClient interface read() method was
>  >  >  >  badly designed; it should have returned a byte array. At present both
>  >  >  >  the client and the sampler have to do String<>byte conversions - and
>  >  >  >  there is no provision for specifying the character set to be used.
>  >  >  >
>  >  >  >  However, that does not affect your current requirements.
>  >  >  >
>  >  >  >  >  Thanks,
>  >  >  >  >  Oghie.
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >
>  >  >  >  >  **********************************************************************
>  >  >  >  >
>  >  >  >  >  E-mail disclaimer
>  >  >  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >  >  >
>  >  >  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >  >  >
>  >  >  >  >  **********************************************************************
>  >  >  >  >
>  >  >  >  >  ---------------------------------------------------------------------
>  >  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >  >
>  >  >  >  >
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

> Yes, but the character set is largely irrelevant, as the hex
> characters are the same in most (all?) sets.

> I was assuming that the hex-encoded binary string would be converted
> to binary in the client. This does not really need a variable encoding
> as hex characters are the same in all main character sets, but for
> portability the client could use a fixed set, e.g. 8859-1 or utf-8.
>
> Likewise for the binary->hex conversion. The client could convert
> binary nibbles to character string and then to bytes, but since we
> know the conversion for these they could be converted direct to bytes.
>
> The character set used to display the response would not matter in this case.

Thanks, needed to be sure we were talking about the same thing and wondered whether hex representation was different according to character set.  

So guess I'm back to what we agreed a few days ago...

Will create client(s) to support hex-binary encoding and length prefix (length of prefix specified in properties for now).  Will create a local version of the TCPClient Interface which you outlined earlier and implement this interface. 

Locally the client will be selectable from the TCP Sampler GUI as you originally suggested.  We need this flexibility across TCP test plans where the same user would wish to run the TCPClientImpl in some and the new client in another without having to change the properties file in-between.     

Can retrofit later if you think we should have a separate TCPSampler/GUI.

> I suppose the question is: are there any protocols where it would be
> difficult/wasteful to return bytes rather than a String?

> If so, then perhaps the SampleResult class needs to be enhanced to
> handle results returned as Strings as well as byte arrays. I'm not
> sure how easy that would be.

> There would also need to be a way to tell the TCP sampler that the
> client returns a String rather than a byte array - assuming we change
> the read() method.

> I need to think some more about this !

Could not come up with a realistic example.

-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 10 October 2008 11:37
To: JMeter Developers List
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > WDYT?
>
>  Overall I think this is a good idea because, if my understanding is correct, it allows the sampler->client->sampler->listener to use distinct character encodings.  Also it puts the encoding logic in the endpoints where it is needed rather than cluttering up the sampler.
>
>  I am a little confused in relation to the encoding/decoding of the hex-encoded binary because there is a double-encoding here as the string containing the hex-encoded binary is itself encoded in a character set.

Yes, but the character set is largely irrelevant, as the hex
characters are the same in most (all?) sets.

>  byte array->hex-encoded binary string->byte array->character set encoded string
>
>  On receiving a response the byte array would need to be converted to a hex-encoded binary string (in the default character set) then back to a byte array (using String.getBytes()) then finally into the target character set.
>
>  I think that you meant the byte array->hex-encoded binary string->byte array conversion to occur within the client and the character set encoding to occur within the listener, is this correct?
>
>

I was assuming that the hex-encoded binary string would be converted
to binary in the client. This does not really need a variable encoding
as hex characters are the same in all main character sets, but for
portability the client could use a fixed set, e.g. 8859-1 or utf-8.

Likewise for the binary->hex conversion. The client could convert
binary nibbles to character string and then to bytes, but since we
know the conversion for these they could be converted direct to bytes.

The character set used to display the response would not matter in this case.

I suppose the question is: are there any protocols where it would be
difficult/wasteful to return bytes rather than a String?

If so, then perhaps the SampleResult class needs to be enhanced to
handle results returned as Strings as well as byte arrays. I'm not
sure how easy that would be.

There would also need to be a way to tell the TCP sampler that the
client returns a String rather than a byte array - assuming we change
the read() method.

I need to think some more about this !

>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 09 October 2008 13:34
>  To: JMeter Developers List
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > > Yes.
>  >
>  >  > I'm still considering how to fix up the read() method. Probably I'll
>  >  > add a readBytes() method and maybe add an abstract class that
>  >  > implements it.
>  >
>  >  > The problem is that the sampler has to convert the string to bytes in
>  >  > order to store it in the sample result. Wire protocols work in terms
>  >  > of bytes, so this means multiple conversions, with the possibility of
>  >  > data loss if some bytes are not both-ways convertible.
>  >
>  >
>  > Am with you now, had thought that strings were stored in the sample result but of course they're not.
>  >
>  >  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>  >
>  >  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?
>
>  Yes, that's possible. In either case existing implementations would
>  need to be amended and rebuilt, so I suppose it does not matter which
>  is done.
>
>  It would be possible to extend the TCPClient interface to add a
>  readBytes() method, and have TCPSampler use instanceof to choose which
>  method to use. This would allow old implementations to continue, but
>  would be less efficient - and we want implementations to return bytes.
>  So I think it would be OK to break compatibility.
>
>  >  Also think that a writeBytes() method is needed.
>  >
>
>  I see what you mean - the client implementation currently has to
>  convert the string into bytes, and the sampler has no control over the
>  encoding that is used.
>
>  I think it's unlikely that there are many 3rd party implementations
>  (if any) so perhaps the string method should be removed.
>
>  We need the input/output encodings to be specified on the sampler GUI
>  (they can default to the current setting, for backward compat.)
>
>  >  > Sorry, should have explained in more detail: I was thinking that setup
>  >  > parameters could be used to configure the protocol implementation.
>  >  > This could be useful for minor variants on the protocol - e.g. 4byte
>  >  > length prefix, and variable length strings.
>  >
>  >
>  > Yes this would make sense.
>  >
>  >  ******
>  >
>  >  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...
>
>  Good idea.
>
>  >  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.
>
>  Yes.
>
>  >  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.
>
>  I was suggesting that the sampler deals only in the data without prefixes.
>  The client needs to know how to add/remove the length prefix.
>  The prefix length is either fixed in the client, or the sampler can
>  pass in a parameter which tells it what particular protocol variation
>  is to be used.
>
>  The sampler is interested in the user data, the client looks after the
>  packaging.
>
>  >  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>  >
>  >  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.
>
>  Yes.
>
>  >  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.
>
>  This will only be true if the input is bytes or a stream.
>  I now think the client show know the encoding (see below).
>
>  >  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.
>
>  I think that depends on how you look at the protocol. In your original
>  case, you would like JMeter to hande hex-encoded binary. You could
>  regard the conversion between hex and binary as part of the protocol,
>  and therefore handled by the client.
>
>  I'm not keen on putting too much processing in the sampler. The idea
>  is that JMeter users can develop their own implementations in order to
>  handle any special processing, and this can be provided in an add-on
>  jar. It's a lot harder for a user to make changes to the sampler.
>
>  I'm hoping that we can come up with more generic sampler which allows
>  client implementations to do what they need to do.
>
>  I'm now beginning to think that the client should still be passed the
>  string from the GUI, but that it also needs the encoding - which it
>  may ignore. This makes it more flexible for clients that need to
>  process the string before transmission.
>
>  >  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.
>
>  Yes, see above.
>
>  >  To achieve this I now think that the following changes should be made:
>  >
>  >  Client:
>  >  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  >  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>  >
>  >  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.
>
>  Yes, that's what I was thinking.
>
>  >  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.
>
>  Yes.
>
>  >  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.
>
>  Agreed.
>
>  >  Sampler GUI:
>  >  From above GUI needs to specify
>  >  a) Length prefix length
>  >  b) Length prefix format OR ClientImpl as above
>
>  No, I think these are part of the client implementation, see above.
>
>  >  In addition the GUI needs to specify
>  >  c) Data format
>
>  Again, I think this is best dealt with by the client implementation.
>
>  >  Sampler:
>  >  The sampler needs to convert the input text string into a byte[] for transmission and a
>  byte[] into a text string (e.g. to hex string for binary/to local
>  character set for character set).  This then goes back into the
>  sampler result using the getBytes() method as before.
>
>  Not quite.
>
>  Originally I thought the sampler should convert the GUI data to bytes
>  using the specified encoding. I now think the sampler should pass in
>  the string and the encoding, as this is more flexible.
>
>  On return, it copies the bytes to the sample result, and sets the
>  appropriate encoding - there is no conversion at that point. The
>  response is only converted to a string when it is extracted from the
>  sample result, e.g. by the Tree View Listener.
>
>  The only issue I can see here is: how does the sampler know what
>  encoding to use for the response? This can be specified on the GUI (or
>  we can assume it is the same as the request encoding). If there is
>  likely to be a possibility that the encoding is defined in the
>  response message (for example, this is how HTTP responses work), then
>  I think it makes sense for the client to extract this information, and
>  pass it back to the sampler somehow.
>
>  We could add a getEncoding() method to the interface, but it might be
>  better to use a generic Object getParameter(String name) to the
>  interface, in case it is later found that there are other items that
>  need to be retrieved. The sampler would need to be updated, but
>  existing clients would not need to be updated.
>
>  In summary, I think the TCPClient interface should probably be amended
>  as follows:
>
>  void write(OutputStream os, String s, String encoding);
>
>  The
>     void write(OutputStream os, InputStream is);
>  method probably does not need the encoding.
>
>  void setParameter(String name, Object value);
>
>  Object getParameter(String name);
>
>  There would be an abstract class that provides a dummy implementation
>  for the get/set Parameter methods.
>
>  WDYT?
>
>  >  ******
>  >
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >
>  > Sent: 08 October 2008 16:57
>  >  To: JMeter Developers List
>  >  Cc: Mike Cronin
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >  >
>  >  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >  >
>  >
>  >  Yes.
>  >
>  >  I'm still considering how to fix up the read() method. Probably I'll
>  >  add a readBytes() method and maybe add an abstract class that
>  >  implements it.
>  >
>  >  The problem is that the sampler has to convert the string to bytes in
>  >  order to store it in the sample result. Wire protocols work in terms
>  >  of bytes, so this means multiple conversions, with the possibility of
>  >  data loss if some bytes are not both-ways convertible.
>  >
>  >  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >  >
>  >
>  >  Sorry, should have explained in more detail: I was thinking that setup
>  >  parameters could be used to configure the protocol implementation.
>  >  This could be useful for minor variants on the protocol - e.g. 4byte
>  >  length prefix, and variable length strings.
>  >
>  >  >
>  >  >  -----Original Message-----
>  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >  Sent: 08 October 2008 13:04
>  >  >  To: JMeter Developers List
>  >  >  Cc: Mike Cronin
>  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  data
>  >  >
>  >  >
>  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  > Hello,
>  >  >  >
>  >  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >  >
>  >  >  Is this a proprietary service, or are there several such services?
>  >  >
>  >  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >  >
>  >  >  >  With this option checked:
>  >  >  >
>  >  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >  >
>  >  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  >  void write(OutputStream os, byte[] b);
>  >  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >  >
>  >  >  >  e.g.
>  >  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >  >
>  >  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >  >
>  >  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >  >
>  >  >
>  >  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >  >
>  >  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  >  a new TCPClient implementation?
>  >  >
>  >  >  At present the TCPSampler only supports a single implementation that
>  >  >  is defined via a property, but it could be enhanced to fetch the name
>  >  >  from the GUI to support test plans which require more than one
>  >  >  implementation.
>  >  >
>  >  >  It might be necessary to extend the TCPClient interface to support
>  >  >  some kind of setup method to allow parameters to be passed in; these
>  >  >  would need to be added to the TCP Sample GUI.
>  >  >
>  >  >  The advantage of this approach would be that there should be no need
>  >  >  to change the TCP Sampler further in order to accommodate new
>  >  >  protocols.
>  >  >
>  >  >  I have just realised that the TCPClient interface read() method was
>  >  >  badly designed; it should have returned a byte array. At present both
>  >  >  the client and the sampler have to do String<>byte conversions - and
>  >  >  there is no provision for specifying the character set to be used.
>  >  >
>  >  >  However, that does not affect your current requirements.
>  >  >
>  >  >  >  Thanks,
>  >  >  >  Oghie.
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >  **********************************************************************
>  >  >  >
>  >  >  >  E-mail disclaimer
>  >  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >  >
>  >  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >  >
>  >  >  >  **********************************************************************
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

Re: TCP Sampler Extension to support length-prefixed binary data

[sebb <se...@gmail.com>]

On 10/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > WDYT?
>
>  Overall I think this is a good idea because, if my understanding is correct, it allows the sampler->client->sampler->listener to use distinct character encodings.  Also it puts the encoding logic in the endpoints where it is needed rather than cluttering up the sampler.
>
>  I am a little confused in relation to the encoding/decoding of the hex-encoded binary because there is a double-encoding here as the string containing the hex-encoded binary is itself encoded in a character set.

Yes, but the character set is largely irrelevant, as the hex
characters are the same in most (all?) sets.

>  byte array->hex-encoded binary string->byte array->character set encoded string
>
>  On receiving a response the byte array would need to be converted to a hex-encoded binary string (in the default character set) then back to a byte array (using String.getBytes()) then finally into the target character set.
>
>  I think that you meant the byte array->hex-encoded binary string->byte array conversion to occur within the client and the character set encoding to occur within the listener, is this correct?
>
>

I was assuming that the hex-encoded binary string would be converted
to binary in the client. This does not really need a variable encoding
as hex characters are the same in all main character sets, but for
portability the client could use a fixed set, e.g. 8859-1 or utf-8.

Likewise for the binary->hex conversion. The client could convert
binary nibbles to character string and then to bytes, but since we
know the conversion for these they could be converted direct to bytes.

The character set used to display the response would not matter in this case.

I suppose the question is: are there any protocols where it would be
difficult/wasteful to return bytes rather than a String?

If so, then perhaps the SampleResult class needs to be enhanced to
handle results returned as Strings as well as byte arrays. I'm not
sure how easy that would be.

There would also need to be a way to tell the TCP sampler that the
client returns a String rather than a byte array - assuming we change
the read() method.

I need to think some more about this !

>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 09 October 2008 13:34
>  To: JMeter Developers List
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > > Yes.
>  >
>  >  > I'm still considering how to fix up the read() method. Probably I'll
>  >  > add a readBytes() method and maybe add an abstract class that
>  >  > implements it.
>  >
>  >  > The problem is that the sampler has to convert the string to bytes in
>  >  > order to store it in the sample result. Wire protocols work in terms
>  >  > of bytes, so this means multiple conversions, with the possibility of
>  >  > data loss if some bytes are not both-ways convertible.
>  >
>  >
>  > Am with you now, had thought that strings were stored in the sample result but of course they're not.
>  >
>  >  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>  >
>  >  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?
>
>  Yes, that's possible. In either case existing implementations would
>  need to be amended and rebuilt, so I suppose it does not matter which
>  is done.
>
>  It would be possible to extend the TCPClient interface to add a
>  readBytes() method, and have TCPSampler use instanceof to choose which
>  method to use. This would allow old implementations to continue, but
>  would be less efficient - and we want implementations to return bytes.
>  So I think it would be OK to break compatibility.
>
>  >  Also think that a writeBytes() method is needed.
>  >
>
>  I see what you mean - the client implementation currently has to
>  convert the string into bytes, and the sampler has no control over the
>  encoding that is used.
>
>  I think it's unlikely that there are many 3rd party implementations
>  (if any) so perhaps the string method should be removed.
>
>  We need the input/output encodings to be specified on the sampler GUI
>  (they can default to the current setting, for backward compat.)
>
>  >  > Sorry, should have explained in more detail: I was thinking that setup
>  >  > parameters could be used to configure the protocol implementation.
>  >  > This could be useful for minor variants on the protocol - e.g. 4byte
>  >  > length prefix, and variable length strings.
>  >
>  >
>  > Yes this would make sense.
>  >
>  >  ******
>  >
>  >  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...
>
>  Good idea.
>
>  >  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.
>
>  Yes.
>
>  >  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.
>
>  I was suggesting that the sampler deals only in the data without prefixes.
>  The client needs to know how to add/remove the length prefix.
>  The prefix length is either fixed in the client, or the sampler can
>  pass in a parameter which tells it what particular protocol variation
>  is to be used.
>
>  The sampler is interested in the user data, the client looks after the
>  packaging.
>
>  >  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>  >
>  >  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.
>
>  Yes.
>
>  >  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.
>
>  This will only be true if the input is bytes or a stream.
>  I now think the client show know the encoding (see below).
>
>  >  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.
>
>  I think that depends on how you look at the protocol. In your original
>  case, you would like JMeter to hande hex-encoded binary. You could
>  regard the conversion between hex and binary as part of the protocol,
>  and therefore handled by the client.
>
>  I'm not keen on putting too much processing in the sampler. The idea
>  is that JMeter users can develop their own implementations in order to
>  handle any special processing, and this can be provided in an add-on
>  jar. It's a lot harder for a user to make changes to the sampler.
>
>  I'm hoping that we can come up with more generic sampler which allows
>  client implementations to do what they need to do.
>
>  I'm now beginning to think that the client should still be passed the
>  string from the GUI, but that it also needs the encoding - which it
>  may ignore. This makes it more flexible for clients that need to
>  process the string before transmission.
>
>  >  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.
>
>  Yes, see above.
>
>  >  To achieve this I now think that the following changes should be made:
>  >
>  >  Client:
>  >  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  >  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>  >
>  >  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.
>
>  Yes, that's what I was thinking.
>
>  >  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.
>
>  Yes.
>
>  >  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.
>
>  Agreed.
>
>  >  Sampler GUI:
>  >  From above GUI needs to specify
>  >  a) Length prefix length
>  >  b) Length prefix format OR ClientImpl as above
>
>  No, I think these are part of the client implementation, see above.
>
>  >  In addition the GUI needs to specify
>  >  c) Data format
>
>  Again, I think this is best dealt with by the client implementation.
>
>  >  Sampler:
>  >  The sampler needs to convert the input text string into a byte[] for transmission and a
>  byte[] into a text string (e.g. to hex string for binary/to local
>  character set for character set).  This then goes back into the
>  sampler result using the getBytes() method as before.
>
>  Not quite.
>
>  Originally I thought the sampler should convert the GUI data to bytes
>  using the specified encoding. I now think the sampler should pass in
>  the string and the encoding, as this is more flexible.
>
>  On return, it copies the bytes to the sample result, and sets the
>  appropriate encoding - there is no conversion at that point. The
>  response is only converted to a string when it is extracted from the
>  sample result, e.g. by the Tree View Listener.
>
>  The only issue I can see here is: how does the sampler know what
>  encoding to use for the response? This can be specified on the GUI (or
>  we can assume it is the same as the request encoding). If there is
>  likely to be a possibility that the encoding is defined in the
>  response message (for example, this is how HTTP responses work), then
>  I think it makes sense for the client to extract this information, and
>  pass it back to the sampler somehow.
>
>  We could add a getEncoding() method to the interface, but it might be
>  better to use a generic Object getParameter(String name) to the
>  interface, in case it is later found that there are other items that
>  need to be retrieved. The sampler would need to be updated, but
>  existing clients would not need to be updated.
>
>  In summary, I think the TCPClient interface should probably be amended
>  as follows:
>
>  void write(OutputStream os, String s, String encoding);
>
>  The
>     void write(OutputStream os, InputStream is);
>  method probably does not need the encoding.
>
>  void setParameter(String name, Object value);
>
>  Object getParameter(String name);
>
>  There would be an abstract class that provides a dummy implementation
>  for the get/set Parameter methods.
>
>  WDYT?
>
>  >  ******
>  >
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >
>  > Sent: 08 October 2008 16:57
>  >  To: JMeter Developers List
>  >  Cc: Mike Cronin
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >  >
>  >  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >  >
>  >
>  >  Yes.
>  >
>  >  I'm still considering how to fix up the read() method. Probably I'll
>  >  add a readBytes() method and maybe add an abstract class that
>  >  implements it.
>  >
>  >  The problem is that the sampler has to convert the string to bytes in
>  >  order to store it in the sample result. Wire protocols work in terms
>  >  of bytes, so this means multiple conversions, with the possibility of
>  >  data loss if some bytes are not both-ways convertible.
>  >
>  >  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >  >
>  >
>  >  Sorry, should have explained in more detail: I was thinking that setup
>  >  parameters could be used to configure the protocol implementation.
>  >  This could be useful for minor variants on the protocol - e.g. 4byte
>  >  length prefix, and variable length strings.
>  >
>  >  >
>  >  >  -----Original Message-----
>  >  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  >  Sent: 08 October 2008 13:04
>  >  >  To: JMeter Developers List
>  >  >  Cc: Mike Cronin
>  >  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  >  data
>  >  >
>  >  >
>  >  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  >  > Hello,
>  >  >  >
>  >  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >  >
>  >  >  Is this a proprietary service, or are there several such services?
>  >  >
>  >  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >  >
>  >  >  >  With this option checked:
>  >  >  >
>  >  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >  >
>  >  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  >  void write(OutputStream os, byte[] b);
>  >  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >  >
>  >  >  >  e.g.
>  >  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >  >
>  >  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >  >
>  >  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >  >
>  >  >
>  >  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >  >
>  >  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  >  a new TCPClient implementation?
>  >  >
>  >  >  At present the TCPSampler only supports a single implementation that
>  >  >  is defined via a property, but it could be enhanced to fetch the name
>  >  >  from the GUI to support test plans which require more than one
>  >  >  implementation.
>  >  >
>  >  >  It might be necessary to extend the TCPClient interface to support
>  >  >  some kind of setup method to allow parameters to be passed in; these
>  >  >  would need to be added to the TCP Sample GUI.
>  >  >
>  >  >  The advantage of this approach would be that there should be no need
>  >  >  to change the TCP Sampler further in order to accommodate new
>  >  >  protocols.
>  >  >
>  >  >  I have just realised that the TCPClient interface read() method was
>  >  >  badly designed; it should have returned a byte array. At present both
>  >  >  the client and the sampler have to do String<>byte conversions - and
>  >  >  there is no provision for specifying the character set to be used.
>  >  >
>  >  >  However, that does not affect your current requirements.
>  >  >
>  >  >  >  Thanks,
>  >  >  >  Oghie.
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >
>  >  >  >  **********************************************************************
>  >  >  >
>  >  >  >  E-mail disclaimer
>  >  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >  >
>  >  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >  >
>  >  >  >  **********************************************************************
>  >  >  >
>  >  >  >  ---------------------------------------------------------------------
>  >  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >  >
>  >  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

> WDYT?

Overall I think this is a good idea because, if my understanding is correct, it allows the sampler->client->sampler->listener to use distinct character encodings.  Also it puts the encoding logic in the endpoints where it is needed rather than cluttering up the sampler.

I am a little confused in relation to the encoding/decoding of the hex-encoded binary because there is a double-encoding here as the string containing the hex-encoded binary is itself encoded in a character set.  

byte array->hex-encoded binary string->byte array->character set encoded string

On receiving a response the byte array would need to be converted to a hex-encoded binary string (in the default character set) then back to a byte array (using String.getBytes()) then finally into the target character set.

I think that you meant the byte array->hex-encoded binary string->byte array conversion to occur within the client and the character set encoding to occur within the listener, is this correct?



-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 09 October 2008 13:34
To: JMeter Developers List
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > Yes.
>
>  > I'm still considering how to fix up the read() method. Probably I'll
>  > add a readBytes() method and maybe add an abstract class that
>  > implements it.
>
>  > The problem is that the sampler has to convert the string to bytes in
>  > order to store it in the sample result. Wire protocols work in terms
>  > of bytes, so this means multiple conversions, with the possibility of
>  > data loss if some bytes are not both-ways convertible.
>
>
> Am with you now, had thought that strings were stored in the sample result but of course they're not.
>
>  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>
>  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?

Yes, that's possible. In either case existing implementations would
need to be amended and rebuilt, so I suppose it does not matter which
is done.

It would be possible to extend the TCPClient interface to add a
readBytes() method, and have TCPSampler use instanceof to choose which
method to use. This would allow old implementations to continue, but
would be less efficient - and we want implementations to return bytes.
So I think it would be OK to break compatibility.

>  Also think that a writeBytes() method is needed.
>

I see what you mean - the client implementation currently has to
convert the string into bytes, and the sampler has no control over the
encoding that is used.

I think it's unlikely that there are many 3rd party implementations
(if any) so perhaps the string method should be removed.

We need the input/output encodings to be specified on the sampler GUI
(they can default to the current setting, for backward compat.)

>  > Sorry, should have explained in more detail: I was thinking that setup
>  > parameters could be used to configure the protocol implementation.
>  > This could be useful for minor variants on the protocol - e.g. 4byte
>  > length prefix, and variable length strings.
>
>
> Yes this would make sense.
>
>  ******
>
>  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...

Good idea.

>  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.

Yes.

>  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.

I was suggesting that the sampler deals only in the data without prefixes.
The client needs to know how to add/remove the length prefix.
The prefix length is either fixed in the client, or the sampler can
pass in a parameter which tells it what particular protocol variation
is to be used.

The sampler is interested in the user data, the client looks after the
packaging.

>  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>
>  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.

Yes.

>  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.

This will only be true if the input is bytes or a stream.
I now think the client show know the encoding (see below).

>  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.

I think that depends on how you look at the protocol. In your original
case, you would like JMeter to hande hex-encoded binary. You could
regard the conversion between hex and binary as part of the protocol,
and therefore handled by the client.

I'm not keen on putting too much processing in the sampler. The idea
is that JMeter users can develop their own implementations in order to
handle any special processing, and this can be provided in an add-on
jar. It's a lot harder for a user to make changes to the sampler.

I'm hoping that we can come up with more generic sampler which allows
client implementations to do what they need to do.

I'm now beginning to think that the client should still be passed the
string from the GUI, but that it also needs the encoding - which it
may ignore. This makes it more flexible for clients that need to
process the string before transmission.

>  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.

Yes, see above.

>  To achieve this I now think that the following changes should be made:
>
>  Client:
>  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>
>  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.

Yes, that's what I was thinking.

>  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.

Yes.

>  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.

Agreed.

>  Sampler GUI:
>  From above GUI needs to specify
>  a) Length prefix length
>  b) Length prefix format OR ClientImpl as above

No, I think these are part of the client implementation, see above.

>  In addition the GUI needs to specify
>  c) Data format

Again, I think this is best dealt with by the client implementation.

>  Sampler:
>  The sampler needs to convert the input text string into a byte[] for transmission and a
byte[] into a text string (e.g. to hex string for binary/to local
character set for character set).  This then goes back into the
sampler result using the getBytes() method as before.

Not quite.

Originally I thought the sampler should convert the GUI data to bytes
using the specified encoding. I now think the sampler should pass in
the string and the encoding, as this is more flexible.

On return, it copies the bytes to the sample result, and sets the
appropriate encoding - there is no conversion at that point. The
response is only converted to a string when it is extracted from the
sample result, e.g. by the Tree View Listener.

The only issue I can see here is: how does the sampler know what
encoding to use for the response? This can be specified on the GUI (or
we can assume it is the same as the request encoding). If there is
likely to be a possibility that the encoding is defined in the
response message (for example, this is how HTTP responses work), then
I think it makes sense for the client to extract this information, and
pass it back to the sampler somehow.

We could add a getEncoding() method to the interface, but it might be
better to use a generic Object getParameter(String name) to the
interface, in case it is later found that there are other items that
need to be retrieved. The sampler would need to be updated, but
existing clients would not need to be updated.

In summary, I think the TCPClient interface should probably be amended
as follows:

void write(OutputStream os, String s, String encoding);

The
    void write(OutputStream os, InputStream is);
method probably does not need the encoding.

void setParameter(String name, Object value);

Object getParameter(String name);

There would be an abstract class that provides a dummy implementation
for the get/set Parameter methods.

WDYT?

>  ******
>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 08 October 2008 16:57
>  To: JMeter Developers List
>  Cc: Mike Cronin
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >
>  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >
>
>  Yes.
>
>  I'm still considering how to fix up the read() method. Probably I'll
>  add a readBytes() method and maybe add an abstract class that
>  implements it.
>
>  The problem is that the sampler has to convert the string to bytes in
>  order to store it in the sample result. Wire protocols work in terms
>  of bytes, so this means multiple conversions, with the possibility of
>  data loss if some bytes are not both-ways convertible.
>
>  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >
>
>  Sorry, should have explained in more detail: I was thinking that setup
>  parameters could be used to configure the protocol implementation.
>  This could be useful for minor variants on the protocol - e.g. 4byte
>  length prefix, and variable length strings.
>
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  Sent: 08 October 2008 13:04
>  >  To: JMeter Developers List
>  >  Cc: Mike Cronin
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > Hello,
>  >  >
>  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >
>  >  Is this a proprietary service, or are there several such services?
>  >
>  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >
>  >  >  With this option checked:
>  >  >
>  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >
>  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  void write(OutputStream os, byte[] b);
>  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >
>  >  >  e.g.
>  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >
>  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >
>  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >
>  >
>  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >
>  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  a new TCPClient implementation?
>  >
>  >  At present the TCPSampler only supports a single implementation that
>  >  is defined via a property, but it could be enhanced to fetch the name
>  >  from the GUI to support test plans which require more than one
>  >  implementation.
>  >
>  >  It might be necessary to extend the TCPClient interface to support
>  >  some kind of setup method to allow parameters to be passed in; these
>  >  would need to be added to the TCP Sample GUI.
>  >
>  >  The advantage of this approach would be that there should be no need
>  >  to change the TCP Sampler further in order to accommodate new
>  >  protocols.
>  >
>  >  I have just realised that the TCPClient interface read() method was
>  >  badly designed; it should have returned a byte array. At present both
>  >  the client and the sampler have to do String<>byte conversions - and
>  >  there is no provision for specifying the character set to be used.
>  >
>  >  However, that does not affect your current requirements.
>  >
>  >  >  Thanks,
>  >  >  Oghie.
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >  **********************************************************************
>  >  >
>  >  >  E-mail disclaimer
>  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >
>  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >
>  >  >  **********************************************************************
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

Re: TCP Sampler Extension to support length-prefixed binary data

[sebb <se...@gmail.com>]

On 09/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> > Yes.
>
>  > I'm still considering how to fix up the read() method. Probably I'll
>  > add a readBytes() method and maybe add an abstract class that
>  > implements it.
>
>  > The problem is that the sampler has to convert the string to bytes in
>  > order to store it in the sample result. Wire protocols work in terms
>  > of bytes, so this means multiple conversions, with the possibility of
>  > data loss if some bytes are not both-ways convertible.
>
>
> Am with you now, had thought that strings were stored in the sample result but of course they're not.
>
>  Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.
>
>  Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these?

Yes, that's possible. In either case existing implementations would
need to be amended and rebuilt, so I suppose it does not matter which
is done.

It would be possible to extend the TCPClient interface to add a
readBytes() method, and have TCPSampler use instanceof to choose which
method to use. This would allow old implementations to continue, but
would be less efficient - and we want implementations to return bytes.
So I think it would be OK to break compatibility.

>  Also think that a writeBytes() method is needed.
>

I see what you mean - the client implementation currently has to
convert the string into bytes, and the sampler has no control over the
encoding that is used.

I think it's unlikely that there are many 3rd party implementations
(if any) so perhaps the string method should be removed.

We need the input/output encodings to be specified on the sampler GUI
(they can default to the current setting, for backward compat.)

>  > Sorry, should have explained in more detail: I was thinking that setup
>  > parameters could be used to configure the protocol implementation.
>  > This could be useful for minor variants on the protocol - e.g. 4byte
>  > length prefix, and variable length strings.
>
>
> Yes this would make sense.
>
>  ******
>
>  Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it...

Good idea.

>  A client extension is required to handle length prefix regardless of whether byte[] or string is returned.

Yes.

>  The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.

I was suggesting that the sampler deals only in the data without prefixes.
The client needs to know how to add/remove the length prefix.
The prefix length is either fixed in the client, or the sampler can
pass in a parameter which tells it what particular protocol variation
is to be used.

The sampler is interested in the user data, the client looks after the
packaging.

>  It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.
>
>  It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters.

Yes.

>  So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.

This will only be true if the input is bytes or a stream.
I now think the client show know the encoding (see below).

>  The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.

I think that depends on how you look at the protocol. In your original
case, you would like JMeter to hande hex-encoded binary. You could
regard the conversion between hex and binary as part of the protocol,
and therefore handled by the client.

I'm not keen on putting too much processing in the sampler. The idea
is that JMeter users can develop their own implementations in order to
handle any special processing, and this can be provided in an add-on
jar. It's a lot harder for a user to make changes to the sampler.

I'm hoping that we can come up with more generic sampler which allows
client implementations to do what they need to do.

I'm now beginning to think that the client should still be passed the
string from the GUI, but that it also needs the encoding - which it
may ignore. This makes it more flexible for clients that need to
process the string before transmission.

>  Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.

Yes, see above.

>  To achieve this I now think that the following changes should be made:
>
>  Client:
>  Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
>  OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.
>
>  Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.

Yes, that's what I was thinking.

>  Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.

Yes.

>  The reason for adding/stripping length bytes is so that the sampler does not have to understand these.

Agreed.

>  Sampler GUI:
>  From above GUI needs to specify
>  a) Length prefix length
>  b) Length prefix format OR ClientImpl as above

No, I think these are part of the client implementation, see above.

>  In addition the GUI needs to specify
>  c) Data format

Again, I think this is best dealt with by the client implementation.

>  Sampler:
>  The sampler needs to convert the input text string into a byte[] for transmission and a
byte[] into a text string (e.g. to hex string for binary/to local
character set for character set).  This then goes back into the
sampler result using the getBytes() method as before.

Not quite.

Originally I thought the sampler should convert the GUI data to bytes
using the specified encoding. I now think the sampler should pass in
the string and the encoding, as this is more flexible.

On return, it copies the bytes to the sample result, and sets the
appropriate encoding - there is no conversion at that point. The
response is only converted to a string when it is extracted from the
sample result, e.g. by the Tree View Listener.

The only issue I can see here is: how does the sampler know what
encoding to use for the response? This can be specified on the GUI (or
we can assume it is the same as the request encoding). If there is
likely to be a possibility that the encoding is defined in the
response message (for example, this is how HTTP responses work), then
I think it makes sense for the client to extract this information, and
pass it back to the sampler somehow.

We could add a getEncoding() method to the interface, but it might be
better to use a generic Object getParameter(String name) to the
interface, in case it is later found that there are other items that
need to be retrieved. The sampler would need to be updated, but
existing clients would not need to be updated.

In summary, I think the TCPClient interface should probably be amended
as follows:

void write(OutputStream os, String s, String encoding);

The
    void write(OutputStream os, InputStream is);
method probably does not need the encoding.

void setParameter(String name, Object value);

Object getParameter(String name);

There would be an abstract class that provides a dummy implementation
for the get/set Parameter methods.

WDYT?

>  ******
>
>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>
> Sent: 08 October 2008 16:57
>  To: JMeter Developers List
>  Cc: Mike Cronin
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>  >
>  >  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>  >
>
>  Yes.
>
>  I'm still considering how to fix up the read() method. Probably I'll
>  add a readBytes() method and maybe add an abstract class that
>  implements it.
>
>  The problem is that the sampler has to convert the string to bytes in
>  order to store it in the sample result. Wire protocols work in terms
>  of bytes, so this means multiple conversions, with the possibility of
>  data loss if some bytes are not both-ways convertible.
>
>  >  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>  >
>
>  Sorry, should have explained in more detail: I was thinking that setup
>  parameters could be used to configure the protocol implementation.
>  This could be useful for minor variants on the protocol - e.g. 4byte
>  length prefix, and variable length strings.
>
>  >
>  >  -----Original Message-----
>  >  From: sebb [mailto:sebbaz@gmail.com]
>  >  Sent: 08 October 2008 13:04
>  >  To: JMeter Developers List
>  >  Cc: Mike Cronin
>  >  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  >  data
>  >
>  >
>  >  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  >  > Hello,
>  >  >
>  >  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>  >
>  >  Is this a proprietary service, or are there several such services?
>  >
>  >  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >  >
>  >  >  With this option checked:
>  >  >
>  >  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >  >
>  >  >  b) The TCPClient has been extended to support the following new methods:
>  >  >  void write(OutputStream os, byte[] b);
>  >  >  byte [] readLengthPrefixed(InputStream is);
>  >  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >  >
>  >  >  e.g.
>  >  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >  >
>  >  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >  >
>  >  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >  >
>  >
>  >  It sounds fairly straightforward, and would not affect existing test plans.
>  >
>  >  I'm just wondering if it might be cleaner to add the functionality as
>  >  a new TCPClient implementation?
>  >
>  >  At present the TCPSampler only supports a single implementation that
>  >  is defined via a property, but it could be enhanced to fetch the name
>  >  from the GUI to support test plans which require more than one
>  >  implementation.
>  >
>  >  It might be necessary to extend the TCPClient interface to support
>  >  some kind of setup method to allow parameters to be passed in; these
>  >  would need to be added to the TCP Sample GUI.
>  >
>  >  The advantage of this approach would be that there should be no need
>  >  to change the TCP Sampler further in order to accommodate new
>  >  protocols.
>  >
>  >  I have just realised that the TCPClient interface read() method was
>  >  badly designed; it should have returned a byte array. At present both
>  >  the client and the sampler have to do String<>byte conversions - and
>  >  there is no provision for specifying the character set to be used.
>  >
>  >  However, that does not affect your current requirements.
>  >
>  >  >  Thanks,
>  >  >  Oghie.
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >
>  >  >  **********************************************************************
>  >  >
>  >  >  E-mail disclaimer
>  >  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >  >
>  >  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >  >
>  >  >  **********************************************************************
>  >  >
>  >  >  ---------------------------------------------------------------------
>  >  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >  >
>  >  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

> Yes.

> I'm still considering how to fix up the read() method. Probably I'll
> add a readBytes() method and maybe add an abstract class that
> implements it.

> The problem is that the sampler has to convert the string to bytes in
> order to store it in the sample result. Wire protocols work in terms
> of bytes, so this means multiple conversions, with the possibility of
> data loss if some bytes are not both-ways convertible.

Am with you now, had thought that strings were stored in the sample result but of course they're not.

Like the idea of adding the readBytes() method and allowing the sampler to convert the bytes as appropriate because the responsibility of the client should be to handle the byte exchange alone.  

Rather than adding the abstract class could the byte[] implementation replace the string read() method in the sampler and drop the byte [] straight into the sampler result?  This would affect existing testplans which are working because of the inherent data conversion and there may not be too many of these? 

Also think that a writeBytes() method is needed.

> Sorry, should have explained in more detail: I was thinking that setup
> parameters could be used to configure the protocol implementation.
> This could be useful for minor variants on the protocol - e.g. 4byte
> length prefix, and variable length strings.

Yes this would make sense.

******

Revisiting the length-prefixed binary extension based on the multiple conversion point raised above, apologies for repetition and stating the obvious here but just need to work through it... 

A client extension is required to handle length prefix regardless of whether byte[] or string is returned.  

The client needs to know how many bytes contain the length and how to interpret these bytes.  This would pollute the client/sampler relationship because the client needs to know about the structure of the byte stream.  

It seems reasonable that a service could use a 2 or 4-byte binary length while the message payload would contain character data so a client and sampler would not necessarily interpret bytes in the same way.  

It also seems reasonable that a length prefix should always specify the no. of bytes to read (as this is a wire-level field) rather than characters. 

So the client should not be concerned about the character set of the data but only by the size and interpretation of the length prefix.  

The sampler needs to know how to represent the data in a human-readable form while not losing any information due to conversion.  Therefore it needs to know whether the data is binary/character set.

Lastly I think it would be justifiable and from a user-perspective preferable to hide the length prefix from sampler input and results.

To achieve this I now think that the following changes should be made:

Client:
Either a single new client TCPLengthPrefixedClientImpl which takes length prefix length and length prefix format binary/character set as constructor/setter arguments.  For this to be flexible these need to provided through the Sampler GUI.
OR multiple new clients BinaryTCPLengthPrefixedClientImpl, DecimalTCPLengthPrefixedClientImpl etc. which take the length prefix length as a constructor.

Client(s) would implement/override writeBytes(byte[]) method and generate the length prefix appropriately based on the size of the byte array.

Client(s) would override readBytes() method and interpret the length bytes from the stream, these would not be passed back to the sampler.

The reason for adding/stripping length bytes is so that the sampler does not have to understand these.  

Sampler GUI:
>From above GUI needs to specify
a) Length prefix length 
b) Length prefix format OR ClientImpl as above

In addition the GUI needs to specify
c) Data format

Sampler:
The sampler needs to convert the input text string into a byte[] for transmission and a byte[] into a text string (e.g. to hex string for binary/to local character set for character set).  This then goes back into the sampler result using the getBytes() method as before.

******

-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 08 October 2008 16:57
To: JMeter Developers List
Cc: Mike Cronin
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>
>  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>

Yes.

I'm still considering how to fix up the read() method. Probably I'll
add a readBytes() method and maybe add an abstract class that
implements it.

The problem is that the sampler has to convert the string to bytes in
order to store it in the sample result. Wire protocols work in terms
of bytes, so this means multiple conversions, with the possibility of
data loss if some bytes are not both-ways convertible.

>  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>

Sorry, should have explained in more detail: I was thinking that setup
parameters could be used to configure the protocol implementation.
This could be useful for minor variants on the protocol - e.g. 4byte
length prefix, and variable length strings.

>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>  Sent: 08 October 2008 13:04
>  To: JMeter Developers List
>  Cc: Mike Cronin
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > Hello,
>  >
>  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>
>  Is this a proprietary service, or are there several such services?
>
>  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >
>  >  With this option checked:
>  >
>  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >
>  >  b) The TCPClient has been extended to support the following new methods:
>  >  void write(OutputStream os, byte[] b);
>  >  byte [] readLengthPrefixed(InputStream is);
>  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >
>  >  e.g.
>  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >
>  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >
>  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >
>
>  It sounds fairly straightforward, and would not affect existing test plans.
>
>  I'm just wondering if it might be cleaner to add the functionality as
>  a new TCPClient implementation?
>
>  At present the TCPSampler only supports a single implementation that
>  is defined via a property, but it could be enhanced to fetch the name
>  from the GUI to support test plans which require more than one
>  implementation.
>
>  It might be necessary to extend the TCPClient interface to support
>  some kind of setup method to allow parameters to be passed in; these
>  would need to be added to the TCP Sample GUI.
>
>  The advantage of this approach would be that there should be no need
>  to change the TCP Sampler further in order to accommodate new
>  protocols.
>
>  I have just realised that the TCPClient interface read() method was
>  badly designed; it should have returned a byte array. At present both
>  the client and the sampler have to do String<>byte conversions - and
>  there is no provision for specifying the character set to be used.
>
>  However, that does not affect your current requirements.
>
>  >  Thanks,
>  >  Oghie.
>  >
>  >
>  >
>  >
>  >
>  >
>  >  **********************************************************************
>  >
>  >  E-mail disclaimer
>  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >
>  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >
>  >  **********************************************************************
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

Re: TCP Sampler Extension to support length-prefixed binary data

[sebb <se...@gmail.com>]

On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.
>
>  This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.
>

Yes.

I'm still considering how to fix up the read() method. Probably I'll
add a readBytes() method and maybe add an abstract class that
implements it.

The problem is that the sampler has to convert the string to bytes in
order to store it in the sample result. Wire protocols work in terms
of bytes, so this means multiple conversions, with the possibility of
data loss if some bytes are not both-ways convertible.

>  Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?
>

Sorry, should have explained in more detail: I was thinking that setup
parameters could be used to configure the protocol implementation.
This could be useful for minor variants on the protocol - e.g. 4byte
length prefix, and variable length strings.

>
>  -----Original Message-----
>  From: sebb [mailto:sebbaz@gmail.com]
>  Sent: 08 October 2008 13:04
>  To: JMeter Developers List
>  Cc: Mike Cronin
>  Subject: Re: TCP Sampler Extension to support length-prefixed binary
>  data
>
>
>  On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
>  > Hello,
>  >
>  >  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.
>
>  Is this a proprietary service, or are there several such services?
>
>  >  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>  >
>  >  With this option checked:
>  >
>  >  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  >  Conversely it reads a byte array and converts it to a hexadecimal string.
>  >
>  >  b) The TCPClient has been extended to support the following new methods:
>  >  void write(OutputStream os, byte[] b);
>  >  byte [] readLengthPrefixed(InputStream is);
>  >  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>  >
>  >  e.g.
>  >  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  >  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>  >
>  >  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>  >
>  >  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>  >
>
>  It sounds fairly straightforward, and would not affect existing test plans.
>
>  I'm just wondering if it might be cleaner to add the functionality as
>  a new TCPClient implementation?
>
>  At present the TCPSampler only supports a single implementation that
>  is defined via a property, but it could be enhanced to fetch the name
>  from the GUI to support test plans which require more than one
>  implementation.
>
>  It might be necessary to extend the TCPClient interface to support
>  some kind of setup method to allow parameters to be passed in; these
>  would need to be added to the TCP Sample GUI.
>
>  The advantage of this approach would be that there should be no need
>  to change the TCP Sampler further in order to accommodate new
>  protocols.
>
>  I have just realised that the TCPClient interface read() method was
>  badly designed; it should have returned a byte array. At present both
>  the client and the sampler have to do String<>byte conversions - and
>  there is no provision for specifying the character set to be used.
>
>  However, that does not affect your current requirements.
>
>  >  Thanks,
>  >  Oghie.
>  >
>  >
>  >
>  >
>  >
>  >
>  >  **********************************************************************
>  >
>  >  E-mail disclaimer
>  >  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>  >
>  >  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>  >
>  >  **********************************************************************
>  >
>  >  ---------------------------------------------------------------------
>  >  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  >  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>  >
>  >
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

Thanks, would be happy to adopt the suggested approach and delegate the protocol-specific conversion to a new TCPClient implementation with selection from the TCPSampler GUI to support test plans using more than one.

This would leave the TCPClient Interface read()/write() method signatures unchanged as strings would be passed in and out.  

Not sure whether setup parameters would be necessary given this approach as TCPClient Implementation would be specific to protocol/character set in use?

-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 08 October 2008 13:04
To: JMeter Developers List
Cc: Mike Cronin
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> Hello,
>
>  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.

Is this a proprietary service, or are there several such services?

>  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>
>  With this option checked:
>
>  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  Conversely it reads a byte array and converts it to a hexadecimal string.
>
>  b) The TCPClient has been extended to support the following new methods:
>  void write(OutputStream os, byte[] b);
>  byte [] readLengthPrefixed(InputStream is);
>  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>
>  e.g.
>  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>
>  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>
>  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>

It sounds fairly straightforward, and would not affect existing test plans.

I'm just wondering if it might be cleaner to add the functionality as
a new TCPClient implementation?

At present the TCPSampler only supports a single implementation that
is defined via a property, but it could be enhanced to fetch the name
from the GUI to support test plans which require more than one
implementation.

It might be necessary to extend the TCPClient interface to support
some kind of setup method to allow parameters to be passed in; these
would need to be added to the TCP Sample GUI.

The advantage of this approach would be that there should be no need
to change the TCP Sampler further in order to accommodate new
protocols.

I have just realised that the TCPClient interface read() method was
badly designed; it should have returned a byte array. At present both
the client and the sampler have to do String<>byte conversions - and
there is no provision for specifying the character set to be used.

However, that does not affect your current requirements.

>  Thanks,
>  Oghie.
>
>
>
>
>
>
>  **********************************************************************
>
>  E-mail disclaimer
>  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>
>  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>
>  **********************************************************************
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

RE: TCP Sampler Extension to support length-prefixed binary data

[Oghie Sheehy <os...@fexcodcc.com>]

Sorry, forgot to answer the question on proprietary service.  

No, not proprietary, message format to ISO8583 standard and length prefix widely used.

-----Original Message-----
From: sebb [mailto:sebbaz@gmail.com]
Sent: 08 October 2008 13:04
To: JMeter Developers List
Cc: Mike Cronin
Subject: Re: TCP Sampler Extension to support length-prefixed binary
data


On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> Hello,
>
>  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.

Is this a proprietary service, or are there several such services?

>  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>
>  With this option checked:
>
>  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  Conversely it reads a byte array and converts it to a hexadecimal string.
>
>  b) The TCPClient has been extended to support the following new methods:
>  void write(OutputStream os, byte[] b);
>  byte [] readLengthPrefixed(InputStream is);
>  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>
>  e.g.
>  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>
>  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>
>  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>

It sounds fairly straightforward, and would not affect existing test plans.

I'm just wondering if it might be cleaner to add the functionality as
a new TCPClient implementation?

At present the TCPSampler only supports a single implementation that
is defined via a property, but it could be enhanced to fetch the name
from the GUI to support test plans which require more than one
implementation.

It might be necessary to extend the TCPClient interface to support
some kind of setup method to allow parameters to be passed in; these
would need to be added to the TCP Sample GUI.

The advantage of this approach would be that there should be no need
to change the TCP Sampler further in order to accommodate new
protocols.

I have just realised that the TCPClient interface read() method was
badly designed; it should have returned a byte array. At present both
the client and the sampler have to do String<>byte conversions - and
there is no provision for specifying the character set to be used.

However, that does not affect your current requirements.

>  Thanks,
>  Oghie.
>
>
>
>
>
>
>  **********************************************************************
>
>  E-mail disclaimer
>  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>
>  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>
>  **********************************************************************
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org

Re: TCP Sampler Extension to support length-prefixed binary data

[sebb <se...@gmail.com>]

On 08/10/2008, Oghie Sheehy <os...@fexcodcc.com> wrote:
> Hello,
>
>  We have a requirement to test a TCP service which expects requests in length-prefixed binary form and responds with data structured in the same way.

Is this a proprietary service, or are there several such services?

>  We have experimentally extended the TCP sampler code to support this by adding a "Length Prefixed Binary Data" option to the GUI.
>
>  With this option checked:
>
>  a) The TCPSampler sample() method treats the input request text as a hexadecimal string and converts it to a byte array.
>  Conversely it reads a byte array and converts it to a hexadecimal string.
>
>  b) The TCPClient has been extended to support the following new methods:
>  void write(OutputStream os, byte[] b);
>  byte [] readLengthPrefixed(InputStream is);
>  The readLengthPrefixed method expects the first 2 bytes from the stream to contain the length (binary) of data to follow and reads this number of subsequent bytes returning a byte array containing both the length and data bytes.
>
>  e.g.
>  Request/Response String: "002d482d2d2d3e454e414320544350495020484541525420424541543c2d2d2d5443504449414c2056312e30312020"
>  Input/Output Byte Array: [0x00,0x2d, 0x48, ......, 0x20]
>
>  This is a simple but limited approach, it does not permit the transmission of non-length prefixed binary data nor does it support length prefixed character data.  Finally as it is implemented the length prefix must be input as two byte binary (input as hexadecimal string) rather than being calculated or input as character decimal.
>
>  There are options to make the solution more flexible but before spending time on these we would like to know whether the minimal approach taken above would be considered for inclusion in a future JMeter release?
>

It sounds fairly straightforward, and would not affect existing test plans.

I'm just wondering if it might be cleaner to add the functionality as
a new TCPClient implementation?

At present the TCPSampler only supports a single implementation that
is defined via a property, but it could be enhanced to fetch the name
from the GUI to support test plans which require more than one
implementation.

It might be necessary to extend the TCPClient interface to support
some kind of setup method to allow parameters to be passed in; these
would need to be added to the TCP Sample GUI.

The advantage of this approach would be that there should be no need
to change the TCP Sampler further in order to accommodate new
protocols.

I have just realised that the TCPClient interface read() method was
badly designed; it should have returned a byte array. At present both
the client and the sampler have to do String<>byte conversions - and
there is no provision for specifying the character set to be used.

However, that does not affect your current requirements.

>  Thanks,
>  Oghie.
>
>
>
>
>
>
>  **********************************************************************
>
>  E-mail disclaimer
>  FEXCO Dynamic Currency Conversion Limited, registered in Ireland, No. 246289. Registered Office: FEXCO Centre, Iveragh Road, Killorglin, Co. Kerry.
>
>  This message, including any attachments, is confidential. If you are not the named recipient, please contact the sender and delete the email from your system.
>
>  **********************************************************************
>
>  ---------------------------------------------------------------------
>  To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
>  For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-dev-help@jakarta.apache.org