search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

I want to replay a Flex session that I recorded with JMeter. When
using multiple users, I get into trouble because all of them use the
same Flex session ID. The requests/responses are AMF3 encoded. With
search-and-replace on the Linux command-line I can easily substitute
the recorded session ID.

I would like to do the same thing dynamically in JMeter somehow. To
make this happen, I need to extract the session ID from the first HTTP
response's binary POST data and somehow store it (for example by
looking for a specific pattern). Then, for every recorded HTTP request
I need to replace the recorded session-ID in the binary POST-data by
the stored session-ID.

Is this possible? Which controllers should I use for this?

Thank you,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi Bruce,

Thanks for your help. It took me almost a full day to get this working
(I am not a Java nor JMeter person).

For anybody else who wants to do the same, below is the full
BeanShell. The script substitutes two regular expression for the
"SDId" (Flex session-id ID) and an application-generated "token" which
are read from a binary AMF3-response body in a Post Processor Regular
Expression (avoiding full AMF3-decoding). The substitutes are done in
binary POST template-files and the results are copied to the
Flex-session/JMeter-thread bound "run/${SDId}/"-directory.

Cheers,
Andrej


import java.util.regex.Pattern;
import java.util.regex.Matcher;
import java.lang.Exception;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.nio.channels.FileChannel;
import java.nio.ByteBuffer;
import org.apache.jmeter.protocol.http.util.HTTPFileArg;
import java.io.StringWriter;
import java.io.PrintWriter;

try
{
  String charsetName = "ISO-8859-1"; // Used for correct conversion
byte[] <-> String.

  HTTPFileArg arg = (HTTPFileArg) sampler.getHTTPFiles()[0];
  String inFilename = arg.getPath();
  if (inFilename.contains("0000_init.amf")) {
     return;  // First AMF-request that generates the DSId, no
substities needed.
  }

  inFilename = inFilename.substring(inFilename.lastIndexOf("/")+1);
  String inPrefix = "requests"; // Directory with AMF template-request.
  String inFullFilename = inPrefix + "/" + inFilename;
  FileInputStream input = new FileInputStream(inFullFilename);
  FileChannel channel = input.getChannel();

  // Map file-contents into byte[].
  int size = (int) channel.size();
  ByteBuffer membuf = channel.map(FileChannel.MapMode.READ_ONLY, 0, size);
  byte[] contents = new byte[size];
  membuf.get(contents, 0, size);

  String DSId = (String) vars.get("DSId");
  String token = (String) vars.get("token");
  String DSIdAndTokenPattern =
"[0-9A-Fa-f]{8}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{4}-[0-9A-Fa-f]{12}";

  Pattern pattern = Pattern.compile(DSIdAndTokenPattern);
  Matcher matcher = pattern.matcher(new String(contents, charsetName));
  StringBuffer sb = new StringBuffer();
  if (DSId != null && matcher.find()) {
    matcher.appendReplacement(sb, DSId);
  }
  else {
    return;
  }

  if (token != null && matcher.find()) {
    matcher.appendReplacement(sb, token);
  }

  matcher.appendTail(sb);

  // Create "run"-directory for this Flex-session: run/${DSId}.
  String outPrefix = "run/" + DSId;
  File dir = new File(outPrefix);
  dir.mkdirs();

  // Write substituted AMF-request to run-directory.
  String outFilename = outPrefix + "/" + inFilename;
  FileOutputStream output = new FileOutputStream(outFilename);
  output.write(sb.toString().getBytes(charsetName));
}
catch (Exception e)
{
  StringWriter writer = new StringWriter();
  e.printStackTrace(new PrintWriter(writer));
  log.error(writer.toString());
}

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Bruce Ide <fl...@gmail.com>]

Every thread gets its own copy of variables, so you don't have to worry
about synchronization for that. It looks like jmeter stores variables in a
thread local hash table or something like that.

In addition, if you use a cookie manager to gather and store a session ID,
this seems to operate on a per-thread basis as well. I'm not entirely
certain if it matters where you define the cookie manager. So essentially in
a regular testing situation, every thread is its own session automatically.

I have a (very rudimentary) semaphore-type plugin and one to enable basic
data sharing between threads. These are stored out on my github repository
if you ever need to do anything like that. I have a feeling that sort of
thing should be used sparingly though; it does make it very hard to look at
the test and understand what's going on in it.

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

I have one more question I was hoping you can answer. When I use a
post-processor Regular Expression Extractor and store the DSId in a
variable, does every user-thread have its own copy of the variable, or
do I need to take care of synchronization? If so, what is the best
practice to do this? I cannot get this from the docs.

Thank you,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,


> If you're doing it in multiple threads you're likely to run into
> synchronization problems unless you put some variable that is unique into
> the filenames that you write out. That's why I suggested writing a new file,
> and consider your original file to be a template.

Yes I understand what you mean, I could use the thread ID too. The
problem is that my binary POST files are created by JMeter while
recoding a session using JMeter as a proxy. It is lots of manual work
to create a template file and update the filename for each sampler in
its corresponding BeanShell pre-processor controller. I have to record
and replay many times so I am looking for a less manual-intensive
manner.

Well, maybe I'll just create the complete JMX file with some scripts.
Synchronization problems seem to inevitable and using a template as
you suggest seems the safest way.

Cheers,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Bruce Ide <fl...@gmail.com>]

If you're doing it in multiple threads you're likely to run into
synchronization problems unless you put some variable that is unique into
the filenames that you write out. That's why I suggested writing a new file,
and consider your original file to be a template.

The code you suggested should work fine. DSId should be unique in each
thread, so you just have one test in a thread group with 1000 users and it
should all just work out. Again, lots of maybes, but give it a try and see
what happens! Jmeter is great for that.

You'll probably want to delete all those files you made at the end of the
test, too...

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi Bruce,

Thanks for your time.

I think I slowly get the idea by your example, but I think I will go
for a BeanShell pre-processor controller anyway. The only thing I am
worried about is multiple threads/users running at the same time. More
specific, in BeanShell (and I guess in Groovy similarly) I am able to
get the configured POST file like this:

HTTPFileArg arg = sampler.getHTTPFiles()[0];

If I update its file path like this:

String newPath = arg.getPath + "_" + ${DSId};
arg.setPath(newPath);

Do I then set the new path for all threads/users or only for the
running thread? I hope the last, but I doubt it.

Thanks again,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Bruce Ide <fl...@gmail.com>]

My example is in groovy. To see if groovy works for you, write a test with a
"View results tree" listener and a BSF sampler with the scripting language
set to "groovy" and the line:

SampleResult.setResponseData("Hello World");

in it. If it works when you run the script  you're good to go. Otherwise if
you want to use groovy to do this, you need to install groovy first. The
instructions at
http://jroller.com/isaaclevin/entry/using_groovy_in_jmeterare what I
used to set jmeter up for this.

Once you've verified that groovy works, reading and writing files is quite
easy. If you create a test file named "C:\test_file" and put in it "This is
a test file" and then create a jmeter test with a BSF sampler that has the
following script in it, a new_test_file should be created with "this is a
blarg file" in it:

String filebytes = new File("C:/test_file").text.replace("test", "blarg");
new File("C:/new_test_file").write(filebytes);
SampleResult.setResponseData(filebytes);

Now the methods in this example _may_ fail, depending on just _how_ binary
your file is, but I think if perl can read it, groovy should be able as
well! If this has trouble, something more advanced may be called for.

Since Groovy is Java (more or less), the "replace" method on the first line
takes a regexp. So you could put your session-id-recognizing regexp as the
first parameter and "${DSid}" as your second parameter and that should work,
assuming the session ID is recognizable in the file. I believe you said you
were already doing this with perl, which is the foundation of my optimism
that this will simply work.

If you need to actually decode the file and then re-encode it, that's beyond
the scope of this particular example.

If you don't want to clobber your template file, I suggest you follow my
example and write a new file which you can then post. If you have many
sessions going at once, you may be able to write the session ID as part of
the new file name (BSF samplers do jmeter variable substitution) and then
post the file with the session ID in your HTTP client request. I'm not sure
if HTTP client requests do variable substitution in the file name field, but
most samplers do so again I'm optimistic that will work.

There are a lot of "ifs" in here, but that's what makes testing fun, isn't
it?

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

>
>> If you have a binary data file on your hard drive and are just sending that
>> file to the server with the post, I think you should be able to open the
>> file with one of the languages supported by the BSF sampler and replace the
>> session ID using a regular expression, and then send the file with an HTTP
>> request.
>

Are there any examples out there that can read read a binary file and
change it before sending it with a HTTP request?

Thank you,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

Thanks for your email.

> If you have a binary data file on your hard drive and are just sending that
> file to the server with the post, I think you should be able to open the
> file with one of the languages supported by the BSF sampler and replace the
> session ID using a regular expression, and then send the file with an HTTP
> request.

This is the exact case that I am facing. The problem is that I have to digg into BSF without being sure wether my intentions can be satisfied. I would just expect there would be a pre-processor controller that would take care of this to avoid a learning curve for BSF. I understand that BSF solves a wider scope of problems, but a pre-processor controller is more user-friendly. 

But anyway, I guess am just wining because of the extra work...

Thanks for your help!
Andrej


---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Bruce Ide <fl...@gmail.com>]

It depends on how you have the data currently. If you can't see the session
ID you want to replace because of the encoding the problem would be rather
complex, but it doesn't sound like that's the problem here.

If you have a binary data file on your hard drive and are just sending that
file to the server with the post, I think you should be able to open the
file with one of the languages supported by the BSF sampler and replace the
session ID using a regular expression, and then send the file with an HTTP
request.

If the data you need to send is already shown as a parameter in the http
request, you can just manually replace the session ID with your ${DSid}
variable and jmeter will do the substitution before it sends the data. I
believe that this is why there's not a regex preprocessor for such requests
-- because normally you would just replace the static data with a variable
in the data stream. Even if the text you want to replace is a substring of a
much larger string to be sent back to the server, this will work.

Sorry if none of that pertains to your problem, I don't know a lot about
flex and the AMF3 encoding side of things, it does sound similar to some of
the web applications I'm testing though.

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

Thank you for your reply.

My case is different from those, unfortunately. I have HTTP POST data recorded in binary files that is sent in the body of each HTTP request. The binary data is an AMF3 encoded Flex message. Each HTTP request sends one binary file as its HTTP request POST body. 

I need to do a search-and-replace in this HTTP request POST body before it is sent to the server. 

I am just amazed that there exist post-processor controllers to extract variables from POST data in an HTTP response by means of regular expressions, but there is no obvious support for modifying the POST data of an HTTP request. I would expect a similar pre-processor controller with match-and-replace regular Perl expressions. 

Or am I missing something?

Thank you,
Andrej
---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org

Re: search-and-replace in binary AMF3 POST data

[Bruce Ide <fl...@gmail.com>]

If you have some nice clean parameters in your HTTP Request, you can just
replace the value of the session ID parameter with ${DSid}. This will
actually work even if there's a long string of junk and you need to put
${DSid} in the middle (I do this with Google Widget tookit RPC requests,
works fine.)

If you need to dynamically compute parameters that you're sending back for
some reason, you can use a BSF preprocessor and use groovy to compute the
string. There's a nifty article at
http://theworkaholic.blogspot.com/2010/03/dynamic-parameters-in-jmeter.html.

-- 
Bruce Ide
FlyingRhenquest@gmail.com

Re: search-and-replace in binary AMF3 POST data

[Andrej van der Zee <an...@gmail.com>]

Hi,

> I need to extract the session ID from the first HTTP
> response's binary POST data and somehow store it (for example by
> looking for a specific pattern).

This was easily done by a post-processor controller of type Regular
Expression Extractor:

Reference Name: DSId
Regular Expression:
.*([[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}).*
Template: $1$
Match No.: 1
Default Value: 0000000-0000-0000-0000-000000000000

> Then, for every recorded HTTP request
> I need to replace the recorded session-ID in the binary POST-data by
> the stored session-ID.

But I cannot find any obvious pre-processor controller to modify POST
data with a regular expression that refers to "DSId". How should I do
this?

Thank you,
Andrej

---------------------------------------------------------------------
To unsubscribe, e-mail: jmeter-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: jmeter-user-help@jakarta.apache.org