You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@jmeter.apache.org by Janusz Kowalczyk <ko...@gmail.com> on 2013/07/08 21:42:01 UTC
How to read a zipped text file and iterate over all the lines
Hi All,
I'd like to create a script that would read a zipped text file, and for
example for each line (passed as variable) send a HTTP request.
So far, I've written a script that works partially :)
Partially, because sending a new line to log.info() prints out this new
line (which is good :) ), but when I set the value of a variable to newly
read line, then the value of that variable remains the same and is always
equal to the first line of that text file.
Please find the code below. To test it:
- create a simple text file with just few lines of random text,
- zip that text file
- Add a JSR223 preprocessor & set the script language to Java
- configure the script accordingly
- add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
- check the msg value of the http request in "View Results Tree"
import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import org.apache.log.Logger;
import org.apache.jmeter.threads.JMeterVariables;
public class JmeterZipReader
{
private String zipFile;
private Logger log;
private String outputVariable;
private JMeterVariables vars;
JmeterZipReader(String zipFile, Logger log, String outputVariable,
JMeterVariables vars){
this.zipFile = zipFile;
this.log = log;
this.outputVariable = outputVariable;
this.vars = vars;
}
// textFile is the name of the text file inside the zip file
public void read(String textFile){
ZipFile zip;
ZipEntry ze;
InputStream input;
BufferedReader br;
try {
zip=new ZipFile(this.zipFile);
ze=zip.getEntry(textFile);
input = zip.getInputStream(ze);
br = new BufferedReader(new InputStreamReader(input,
"US-ASCII"));
String line;
int cnt = 0;
while((line = br.readLine()) != null && cnt <= 10) {
this.vars.put(this.outputVariable, line);
log.info(line);
cnt += 1;
}
br.close();
zip.close();
}
catch (Exception e) {
log.error("Unhandled exception:");
e.printStackTrace();
}
}
}
JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
"CONTEXT", ctx.getVariables());
jzr.read("test.csv");
Many thanks,
Janusz
Re: How to read a zipped text file and iterate over all the lines
Posted by sebb <se...@gmail.com>.
As already noted:
Both CSV DataSet and StringFromFile have to handle repeated reads from
the same file.
They do this without closing and re-opening (except at EOF).
It should be simple to amend either or both for your needs; just
replace the file IO with zip file IO.
Apache Commons Compress has Zip code if you need it.
On 10 July 2013 11:31, Janusz Kowalczyk <ko...@gmail.com> wrote:
> btw. I know that closing & reopening a BufferedReader for the same file is
> lame, but I don't have enough time to optimize it :)
>
>
> On 10 July 2013 11:28, Janusz Kowalczyk <ko...@gmail.com> wrote:
>
>> OK, I solved this problem by:
>> adding "Only once controller" to the thread group
>> putting JSR223 Sampler to that controller
>> adding a "ForEach Controller" after that controller and iterating over all
>> the "CONTEXT" variables.
>>
>> "ForEach Controller" configuration looks like:
>> Input variable prefix: CONTEXT
>> Start index for loop: 0
>> End index for loop: ${LINE_CNT}
>> Output variable name: CONTEXT
>> Select "Add "_" before number?"
>>
>>
>> One thing to remember is that when the txt file inside the zip archive is
>> big, then you need to increase the heap size!
>>
>> *And here's the code:*
>> import java.io.*;
>> import java.util.zip.ZipEntry;
>> import java.util.zip.ZipFile;
>> import org.apache.log.Logger;
>> import org.apache.jmeter.threads.JMeterVariables;
>> import org.apache.jmeter.util.JMeterUtils;
>> import java.io.IOException;
>> import org.apache.jmeter.services.FileServer;
>>
>> /**
>> * A Class that reads text file inside a zip archive and sets the as many
>> "CONTEXT_{x}" variables as the number of lines in the text file.
>> *
>> * Author: Janusz Kowalczyk
>> * Created: 2013-07-09
>> **/
>> public class JmeterZipReader {
>> private String zipFile;
>> private Logger log;
>> private JMeterVariables vars;
>>
>> JmeterZipReader(String zipFile, Logger log, JMeterVariables vars){
>> this.zipFile = zipFile;
>> this.log = log;
>> this.vars = vars;
>> File f = new File(zipFile);
>> if( !f.exists()) {
>> log.error("FILE '"+zipFile+"' DOESN'T EXISTS!!!");
>> throw new IOException("FILE '"+zipFile+"' DOESN'T
>> EXISTS!!!");
>> }
>> }
>>
>> public void read(String textFile){
>> ZipFile zip;
>> ZipEntry ze;
>> InputStream input;
>> BufferedReader br;
>>
>> try {
>> log.info("\n/******************************************\nOpening
>> file " + textFile + " from zip archive: " + this.zipFile +
>> "\n******************************************/");
>> zip=new ZipFile(this.zipFile);
>> ze=zip.getEntry(textFile);
>> input = zip.getInputStream(ze);
>> br = new BufferedReader(new InputStreamReader(input,
>> "US-ASCII"));
>> log.info("Reading file: " + textFile);
>>
>> int LINE_CNT = 0;
>> while (br.readLine() != null) LINE_CNT++;
>> // you can limit the number of rows to a specific number
>> // to avoid problems with the heap space
>> //if (LINE_CNT >= 35000) {
>> // LINE_CNT = 35000;
>> //}
>> vars.put("LINE_CNT", Integer.toString(LINE_CNT));
>> br.close();
>> zip.close();
>> log.info("\n/******************************************\ntext
>> file inside the zip archive has: " + LINE_CNT + "
>> lines!!!\n******************************************/");
>>
>>
>> log.info("\n/******************************************\nReopeing
>> the zip and csv file for setting up all the
>> variables!!\n******************************************/");
>> zip=new ZipFile(this.zipFile);
>> ze=zip.getEntry(textFile);
>> input = zip.getInputStream(ze);
>> br = new BufferedReader(new InputStreamReader(input,
>> "US-ASCII"));
>> String line;
>> for (int i=0; i < LINE_CNT; i++) {
>> line = br.readLine();
>> vars.put("CONTEXT_"+i, line);
>> }
>> br.close();
>> zip.close();
>> log.info("\n/******************************************\nZIP
>> & CSV files were closed!!\n******************************************/");
>> }
>> catch (Exception e) {
>> log.error("Unhandled exception:");
>> e.printStackTrace();
>> }
>> }
>> }
>>
>> String zipFilePath = FileServer.getFileServer().getBaseDir() +
>> "/data/your_zipped_txt_file.zip";
>> JmeterZipReader jzr = new JmeterZipReader(zipFilePath, log,
>> ctx.getVariables());
>> jzr.read("name_of_the_txt_file_inside_the_zip_file.txt");
>>
>>
>> Cheers,
>> Janusz
>>
>>
>> On 9 July 2013 19:37, Deepak Shetty <sh...@gmail.com> wrote:
>>
>>> >that I don't wan't to unzip those files,
>>> The question is why do you want to do that? whats the benefit? - Again you
>>> have setup and tear down threadgroups so you can do it before your test
>>> runs and clean it up after its done (or as part of your build) - note you
>>> have to factor in multiple threads too ..
>>>
>>>
>>> On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
>>> <ko...@gmail.com>wrote:
>>>
>>> > The thing is that I don't wan't to unzip those files, but just read
>>> > directly from them.
>>> >
>>> > OK, maybe easier way would be just returning a list or a map of
>>> variables,
>>> > so that I could iterate over it?
>>> >
>>> > ps. I put this strange loop just for the sake of the demo, so it doesn't
>>> > read more that 10 lines at a time :)
>>> >
>>> >
>>> >
>>> > On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
>>> >
>>> > > this.vars.put(this.outputVariable, line); ==> Always overwrites the
>>> > current
>>> > > value
>>> > > should be the last line however unless your loop is also incorrect
>>> > >
>>> > > But the better way is to get your data in the form you want first ,
>>> then
>>> > > start the test rather than the test writing some code to read the data
>>> > i.e.
>>> > > unzip your files whatever before you mai n test begins (as part of
>>> your
>>> > > build or as a startup threadgroup )
>>> > >
>>> > >
>>> > > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
>>> > > kowalczykjanusz@gmail.com
>>> > > > wrote:
>>> > >
>>> > > > Hi All,
>>> > > >
>>> > > > I'd like to create a script that would read a zipped text file, and
>>> for
>>> > > > example for each line (passed as variable) send a HTTP request.
>>> > > >
>>> > > > So far, I've written a script that works partially :)
>>> > > > Partially, because sending a new line to log.info() prints out this
>>> > new
>>> > > > line (which is good :) ), but when I set the value of a variable to
>>> > newly
>>> > > > read line, then the value of that variable remains the same and is
>>> > always
>>> > > > equal to the first line of that text file.
>>> > > >
>>> > > > Please find the code below. To test it:
>>> > > > - create a simple text file with just few lines of random text,
>>> > > > - zip that text file
>>> > > > - Add a JSR223 preprocessor & set the script language to Java
>>> > > > - configure the script accordingly
>>> > > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
>>> > > > - check the msg value of the http request in "View Results Tree"
>>> > > >
>>> > > >
>>> > > > import java.io.*;
>>> > > > import java.util.zip.ZipEntry;
>>> > > > import java.util.zip.ZipFile;
>>> > > > import org.apache.log.Logger;
>>> > > > import org.apache.jmeter.threads.JMeterVariables;
>>> > > >
>>> > > > public class JmeterZipReader
>>> > > > {
>>> > > > private String zipFile;
>>> > > > private Logger log;
>>> > > > private String outputVariable;
>>> > > > private JMeterVariables vars;
>>> > > >
>>> > > > JmeterZipReader(String zipFile, Logger log, String
>>> outputVariable,
>>> > > > JMeterVariables vars){
>>> > > > this.zipFile = zipFile;
>>> > > > this.log = log;
>>> > > > this.outputVariable = outputVariable;
>>> > > > this.vars = vars;
>>> > > > }
>>> > > >
>>> > > > // textFile is the name of the text file inside the zip file
>>> > > > public void read(String textFile){
>>> > > > ZipFile zip;
>>> > > > ZipEntry ze;
>>> > > > InputStream input;
>>> > > > BufferedReader br;
>>> > > >
>>> > > > try {
>>> > > > zip=new ZipFile(this.zipFile);
>>> > > > ze=zip.getEntry(textFile);
>>> > > > input = zip.getInputStream(ze);
>>> > > > br = new BufferedReader(new InputStreamReader(input,
>>> > > > "US-ASCII"));
>>> > > >
>>> > > > String line;
>>> > > > int cnt = 0;
>>> > > > while((line = br.readLine()) != null && cnt <= 10) {
>>> > > > this.vars.put(this.outputVariable, line);
>>> > > > log.info(line);
>>> > > > cnt += 1;
>>> > > > }
>>> > > >
>>> > > > br.close();
>>> > > > zip.close();
>>> > > > }
>>> > > > catch (Exception e) {
>>> > > > log.error("Unhandled exception:");
>>> > > > e.printStackTrace();
>>> > > > }
>>> > > > }
>>> > > > }
>>> > > >
>>> > > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip",
>>> log,
>>> > > > "CONTEXT", ctx.getVariables());
>>> > > > jzr.read("test.csv");
>>> > > >
>>> > > >
>>> > > >
>>> > > > Many thanks,
>>> > > > Janusz
>>> > > >
>>> > >
>>> >
>>>
>>
>>
>>
>>
>
>
> --
> Zapraszam na swojego foto-bloga: http://na100procentchyba.wordpress.com/
> Autopodpis: Staraj się używać pola Ukryty do Wiadomości (UDW) przy
> wysyłaniu wiadomości do wielu odbiorców, ograniczysz przez to
> rozprzestrzenianie się spamu!
> Autosignature: Try to use field BCC (blind carbon copy) when sending
> message to many recepients, it will restrain spread of spam!
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: How to read a zipped text file and iterate over all the lines
Posted by Janusz Kowalczyk <ko...@gmail.com>.
btw. I know that closing & reopening a BufferedReader for the same file is
lame, but I don't have enough time to optimize it :)
On 10 July 2013 11:28, Janusz Kowalczyk <ko...@gmail.com> wrote:
> OK, I solved this problem by:
> adding "Only once controller" to the thread group
> putting JSR223 Sampler to that controller
> adding a "ForEach Controller" after that controller and iterating over all
> the "CONTEXT" variables.
>
> "ForEach Controller" configuration looks like:
> Input variable prefix: CONTEXT
> Start index for loop: 0
> End index for loop: ${LINE_CNT}
> Output variable name: CONTEXT
> Select "Add "_" before number?"
>
>
> One thing to remember is that when the txt file inside the zip archive is
> big, then you need to increase the heap size!
>
> *And here's the code:*
> import java.io.*;
> import java.util.zip.ZipEntry;
> import java.util.zip.ZipFile;
> import org.apache.log.Logger;
> import org.apache.jmeter.threads.JMeterVariables;
> import org.apache.jmeter.util.JMeterUtils;
> import java.io.IOException;
> import org.apache.jmeter.services.FileServer;
>
> /**
> * A Class that reads text file inside a zip archive and sets the as many
> "CONTEXT_{x}" variables as the number of lines in the text file.
> *
> * Author: Janusz Kowalczyk
> * Created: 2013-07-09
> **/
> public class JmeterZipReader {
> private String zipFile;
> private Logger log;
> private JMeterVariables vars;
>
> JmeterZipReader(String zipFile, Logger log, JMeterVariables vars){
> this.zipFile = zipFile;
> this.log = log;
> this.vars = vars;
> File f = new File(zipFile);
> if( !f.exists()) {
> log.error("FILE '"+zipFile+"' DOESN'T EXISTS!!!");
> throw new IOException("FILE '"+zipFile+"' DOESN'T
> EXISTS!!!");
> }
> }
>
> public void read(String textFile){
> ZipFile zip;
> ZipEntry ze;
> InputStream input;
> BufferedReader br;
>
> try {
> log.info("\n/******************************************\nOpening
> file " + textFile + " from zip archive: " + this.zipFile +
> "\n******************************************/");
> zip=new ZipFile(this.zipFile);
> ze=zip.getEntry(textFile);
> input = zip.getInputStream(ze);
> br = new BufferedReader(new InputStreamReader(input,
> "US-ASCII"));
> log.info("Reading file: " + textFile);
>
> int LINE_CNT = 0;
> while (br.readLine() != null) LINE_CNT++;
> // you can limit the number of rows to a specific number
> // to avoid problems with the heap space
> //if (LINE_CNT >= 35000) {
> // LINE_CNT = 35000;
> //}
> vars.put("LINE_CNT", Integer.toString(LINE_CNT));
> br.close();
> zip.close();
> log.info("\n/******************************************\ntext
> file inside the zip archive has: " + LINE_CNT + "
> lines!!!\n******************************************/");
>
>
> log.info("\n/******************************************\nReopeing
> the zip and csv file for setting up all the
> variables!!\n******************************************/");
> zip=new ZipFile(this.zipFile);
> ze=zip.getEntry(textFile);
> input = zip.getInputStream(ze);
> br = new BufferedReader(new InputStreamReader(input,
> "US-ASCII"));
> String line;
> for (int i=0; i < LINE_CNT; i++) {
> line = br.readLine();
> vars.put("CONTEXT_"+i, line);
> }
> br.close();
> zip.close();
> log.info("\n/******************************************\nZIP
> & CSV files were closed!!\n******************************************/");
> }
> catch (Exception e) {
> log.error("Unhandled exception:");
> e.printStackTrace();
> }
> }
> }
>
> String zipFilePath = FileServer.getFileServer().getBaseDir() +
> "/data/your_zipped_txt_file.zip";
> JmeterZipReader jzr = new JmeterZipReader(zipFilePath, log,
> ctx.getVariables());
> jzr.read("name_of_the_txt_file_inside_the_zip_file.txt");
>
>
> Cheers,
> Janusz
>
>
> On 9 July 2013 19:37, Deepak Shetty <sh...@gmail.com> wrote:
>
>> >that I don't wan't to unzip those files,
>> The question is why do you want to do that? whats the benefit? - Again you
>> have setup and tear down threadgroups so you can do it before your test
>> runs and clean it up after its done (or as part of your build) - note you
>> have to factor in multiple threads too ..
>>
>>
>> On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
>> <ko...@gmail.com>wrote:
>>
>> > The thing is that I don't wan't to unzip those files, but just read
>> > directly from them.
>> >
>> > OK, maybe easier way would be just returning a list or a map of
>> variables,
>> > so that I could iterate over it?
>> >
>> > ps. I put this strange loop just for the sake of the demo, so it doesn't
>> > read more that 10 lines at a time :)
>> >
>> >
>> >
>> > On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
>> >
>> > > this.vars.put(this.outputVariable, line); ==> Always overwrites the
>> > current
>> > > value
>> > > should be the last line however unless your loop is also incorrect
>> > >
>> > > But the better way is to get your data in the form you want first ,
>> then
>> > > start the test rather than the test writing some code to read the data
>> > i.e.
>> > > unzip your files whatever before you mai n test begins (as part of
>> your
>> > > build or as a startup threadgroup )
>> > >
>> > >
>> > > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
>> > > kowalczykjanusz@gmail.com
>> > > > wrote:
>> > >
>> > > > Hi All,
>> > > >
>> > > > I'd like to create a script that would read a zipped text file, and
>> for
>> > > > example for each line (passed as variable) send a HTTP request.
>> > > >
>> > > > So far, I've written a script that works partially :)
>> > > > Partially, because sending a new line to log.info() prints out this
>> > new
>> > > > line (which is good :) ), but when I set the value of a variable to
>> > newly
>> > > > read line, then the value of that variable remains the same and is
>> > always
>> > > > equal to the first line of that text file.
>> > > >
>> > > > Please find the code below. To test it:
>> > > > - create a simple text file with just few lines of random text,
>> > > > - zip that text file
>> > > > - Add a JSR223 preprocessor & set the script language to Java
>> > > > - configure the script accordingly
>> > > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
>> > > > - check the msg value of the http request in "View Results Tree"
>> > > >
>> > > >
>> > > > import java.io.*;
>> > > > import java.util.zip.ZipEntry;
>> > > > import java.util.zip.ZipFile;
>> > > > import org.apache.log.Logger;
>> > > > import org.apache.jmeter.threads.JMeterVariables;
>> > > >
>> > > > public class JmeterZipReader
>> > > > {
>> > > > private String zipFile;
>> > > > private Logger log;
>> > > > private String outputVariable;
>> > > > private JMeterVariables vars;
>> > > >
>> > > > JmeterZipReader(String zipFile, Logger log, String
>> outputVariable,
>> > > > JMeterVariables vars){
>> > > > this.zipFile = zipFile;
>> > > > this.log = log;
>> > > > this.outputVariable = outputVariable;
>> > > > this.vars = vars;
>> > > > }
>> > > >
>> > > > // textFile is the name of the text file inside the zip file
>> > > > public void read(String textFile){
>> > > > ZipFile zip;
>> > > > ZipEntry ze;
>> > > > InputStream input;
>> > > > BufferedReader br;
>> > > >
>> > > > try {
>> > > > zip=new ZipFile(this.zipFile);
>> > > > ze=zip.getEntry(textFile);
>> > > > input = zip.getInputStream(ze);
>> > > > br = new BufferedReader(new InputStreamReader(input,
>> > > > "US-ASCII"));
>> > > >
>> > > > String line;
>> > > > int cnt = 0;
>> > > > while((line = br.readLine()) != null && cnt <= 10) {
>> > > > this.vars.put(this.outputVariable, line);
>> > > > log.info(line);
>> > > > cnt += 1;
>> > > > }
>> > > >
>> > > > br.close();
>> > > > zip.close();
>> > > > }
>> > > > catch (Exception e) {
>> > > > log.error("Unhandled exception:");
>> > > > e.printStackTrace();
>> > > > }
>> > > > }
>> > > > }
>> > > >
>> > > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip",
>> log,
>> > > > "CONTEXT", ctx.getVariables());
>> > > > jzr.read("test.csv");
>> > > >
>> > > >
>> > > >
>> > > > Many thanks,
>> > > > Janusz
>> > > >
>> > >
>> >
>>
>
>
>
>
--
Zapraszam na swojego foto-bloga: http://na100procentchyba.wordpress.com/
Autopodpis: Staraj się używać pola Ukryty do Wiadomości (UDW) przy
wysyłaniu wiadomości do wielu odbiorców, ograniczysz przez to
rozprzestrzenianie się spamu!
Autosignature: Try to use field BCC (blind carbon copy) when sending
message to many recepients, it will restrain spread of spam!
Re: How to read a zipped text file and iterate over all the lines
Posted by Janusz Kowalczyk <ko...@gmail.com>.
OK, I solved this problem by:
adding "Only once controller" to the thread group
putting JSR223 Sampler to that controller
adding a "ForEach Controller" after that controller and iterating over all
the "CONTEXT" variables.
"ForEach Controller" configuration looks like:
Input variable prefix: CONTEXT
Start index for loop: 0
End index for loop: ${LINE_CNT}
Output variable name: CONTEXT
Select "Add "_" before number?"
One thing to remember is that when the txt file inside the zip archive is
big, then you need to increase the heap size!
*And here's the code:*
import java.io.*;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import org.apache.log.Logger;
import org.apache.jmeter.threads.JMeterVariables;
import org.apache.jmeter.util.JMeterUtils;
import java.io.IOException;
import org.apache.jmeter.services.FileServer;
/**
* A Class that reads text file inside a zip archive and sets the as many
"CONTEXT_{x}" variables as the number of lines in the text file.
*
* Author: Janusz Kowalczyk
* Created: 2013-07-09
**/
public class JmeterZipReader {
private String zipFile;
private Logger log;
private JMeterVariables vars;
JmeterZipReader(String zipFile, Logger log, JMeterVariables vars){
this.zipFile = zipFile;
this.log = log;
this.vars = vars;
File f = new File(zipFile);
if( !f.exists()) {
log.error("FILE '"+zipFile+"' DOESN'T EXISTS!!!");
throw new IOException("FILE '"+zipFile+"' DOESN'T
EXISTS!!!");
}
}
public void read(String textFile){
ZipFile zip;
ZipEntry ze;
InputStream input;
BufferedReader br;
try {
log.info("\n/******************************************\nOpening
file " + textFile + " from zip archive: " + this.zipFile +
"\n******************************************/");
zip=new ZipFile(this.zipFile);
ze=zip.getEntry(textFile);
input = zip.getInputStream(ze);
br = new BufferedReader(new InputStreamReader(input,
"US-ASCII"));
log.info("Reading file: " + textFile);
int LINE_CNT = 0;
while (br.readLine() != null) LINE_CNT++;
// you can limit the number of rows to a specific number
// to avoid problems with the heap space
//if (LINE_CNT >= 35000) {
// LINE_CNT = 35000;
//}
vars.put("LINE_CNT", Integer.toString(LINE_CNT));
br.close();
zip.close();
log.info("\n/******************************************\ntext
file inside the zip archive has: " + LINE_CNT + "
lines!!!\n******************************************/");
log.info("\n/******************************************\nReopeing
the zip and csv file for setting up all the
variables!!\n******************************************/");
zip=new ZipFile(this.zipFile);
ze=zip.getEntry(textFile);
input = zip.getInputStream(ze);
br = new BufferedReader(new InputStreamReader(input,
"US-ASCII"));
String line;
for (int i=0; i < LINE_CNT; i++) {
line = br.readLine();
vars.put("CONTEXT_"+i, line);
}
br.close();
zip.close();
log.info("\n/******************************************\nZIP &
CSV files were closed!!\n******************************************/");
}
catch (Exception e) {
log.error("Unhandled exception:");
e.printStackTrace();
}
}
}
String zipFilePath = FileServer.getFileServer().getBaseDir() +
"/data/your_zipped_txt_file.zip";
JmeterZipReader jzr = new JmeterZipReader(zipFilePath, log,
ctx.getVariables());
jzr.read("name_of_the_txt_file_inside_the_zip_file.txt");
Cheers,
Janusz
On 9 July 2013 19:37, Deepak Shetty <sh...@gmail.com> wrote:
> >that I don't wan't to unzip those files,
> The question is why do you want to do that? whats the benefit? - Again you
> have setup and tear down threadgroups so you can do it before your test
> runs and clean it up after its done (or as part of your build) - note you
> have to factor in multiple threads too ..
>
>
> On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
> <ko...@gmail.com>wrote:
>
> > The thing is that I don't wan't to unzip those files, but just read
> > directly from them.
> >
> > OK, maybe easier way would be just returning a list or a map of
> variables,
> > so that I could iterate over it?
> >
> > ps. I put this strange loop just for the sake of the demo, so it doesn't
> > read more that 10 lines at a time :)
> >
> >
> >
> > On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
> >
> > > this.vars.put(this.outputVariable, line); ==> Always overwrites the
> > current
> > > value
> > > should be the last line however unless your loop is also incorrect
> > >
> > > But the better way is to get your data in the form you want first ,
> then
> > > start the test rather than the test writing some code to read the data
> > i.e.
> > > unzip your files whatever before you mai n test begins (as part of your
> > > build or as a startup threadgroup )
> > >
> > >
> > > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
> > > kowalczykjanusz@gmail.com
> > > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to create a script that would read a zipped text file, and
> for
> > > > example for each line (passed as variable) send a HTTP request.
> > > >
> > > > So far, I've written a script that works partially :)
> > > > Partially, because sending a new line to log.info() prints out this
> > new
> > > > line (which is good :) ), but when I set the value of a variable to
> > newly
> > > > read line, then the value of that variable remains the same and is
> > always
> > > > equal to the first line of that text file.
> > > >
> > > > Please find the code below. To test it:
> > > > - create a simple text file with just few lines of random text,
> > > > - zip that text file
> > > > - Add a JSR223 preprocessor & set the script language to Java
> > > > - configure the script accordingly
> > > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
> > > > - check the msg value of the http request in "View Results Tree"
> > > >
> > > >
> > > > import java.io.*;
> > > > import java.util.zip.ZipEntry;
> > > > import java.util.zip.ZipFile;
> > > > import org.apache.log.Logger;
> > > > import org.apache.jmeter.threads.JMeterVariables;
> > > >
> > > > public class JmeterZipReader
> > > > {
> > > > private String zipFile;
> > > > private Logger log;
> > > > private String outputVariable;
> > > > private JMeterVariables vars;
> > > >
> > > > JmeterZipReader(String zipFile, Logger log, String
> outputVariable,
> > > > JMeterVariables vars){
> > > > this.zipFile = zipFile;
> > > > this.log = log;
> > > > this.outputVariable = outputVariable;
> > > > this.vars = vars;
> > > > }
> > > >
> > > > // textFile is the name of the text file inside the zip file
> > > > public void read(String textFile){
> > > > ZipFile zip;
> > > > ZipEntry ze;
> > > > InputStream input;
> > > > BufferedReader br;
> > > >
> > > > try {
> > > > zip=new ZipFile(this.zipFile);
> > > > ze=zip.getEntry(textFile);
> > > > input = zip.getInputStream(ze);
> > > > br = new BufferedReader(new InputStreamReader(input,
> > > > "US-ASCII"));
> > > >
> > > > String line;
> > > > int cnt = 0;
> > > > while((line = br.readLine()) != null && cnt <= 10) {
> > > > this.vars.put(this.outputVariable, line);
> > > > log.info(line);
> > > > cnt += 1;
> > > > }
> > > >
> > > > br.close();
> > > > zip.close();
> > > > }
> > > > catch (Exception e) {
> > > > log.error("Unhandled exception:");
> > > > e.printStackTrace();
> > > > }
> > > > }
> > > > }
> > > >
> > > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
> > > > "CONTEXT", ctx.getVariables());
> > > > jzr.read("test.csv");
> > > >
> > > >
> > > >
> > > > Many thanks,
> > > > Janusz
> > > >
> > >
> >
>
Re: How to read a zipped text file and iterate over all the lines
Posted by Janusz Kowalczyk <ko...@gmail.com>.
Ah bugger, I almost forgot to answer your question regarding the benefits.
The reason is that git doesn't work well with large files (i.e. 250MB data
files).
After zipping, they shrink to 10-15MB and git handle them quite nicely.
Second thing is that when we launch such a test on Jenkins, we don't have
to worry to unpack & delete :)
Third reason is "why not?" :) You never know when such a code will come
handy in different project :)
On 9 July 2013 19:37, Deepak Shetty <sh...@gmail.com> wrote:
> >that I don't wan't to unzip those files,
> The question is why do you want to do that? whats the benefit? - Again you
> have setup and tear down threadgroups so you can do it before your test
> runs and clean it up after its done (or as part of your build) - note you
> have to factor in multiple threads too ..
>
>
> On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
> <ko...@gmail.com>wrote:
>
> > The thing is that I don't wan't to unzip those files, but just read
> > directly from them.
> >
> > OK, maybe easier way would be just returning a list or a map of
> variables,
> > so that I could iterate over it?
> >
> > ps. I put this strange loop just for the sake of the demo, so it doesn't
> > read more that 10 lines at a time :)
> >
> >
> >
> > On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
> >
> > > this.vars.put(this.outputVariable, line); ==> Always overwrites the
> > current
> > > value
> > > should be the last line however unless your loop is also incorrect
> > >
> > > But the better way is to get your data in the form you want first ,
> then
> > > start the test rather than the test writing some code to read the data
> > i.e.
> > > unzip your files whatever before you mai n test begins (as part of your
> > > build or as a startup threadgroup )
> > >
> > >
> > > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
> > > kowalczykjanusz@gmail.com
> > > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I'd like to create a script that would read a zipped text file, and
> for
> > > > example for each line (passed as variable) send a HTTP request.
> > > >
> > > > So far, I've written a script that works partially :)
> > > > Partially, because sending a new line to log.info() prints out this
> > new
> > > > line (which is good :) ), but when I set the value of a variable to
> > newly
> > > > read line, then the value of that variable remains the same and is
> > always
> > > > equal to the first line of that text file.
> > > >
> > > > Please find the code below. To test it:
> > > > - create a simple text file with just few lines of random text,
> > > > - zip that text file
> > > > - Add a JSR223 preprocessor & set the script language to Java
> > > > - configure the script accordingly
> > > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
> > > > - check the msg value of the http request in "View Results Tree"
> > > >
> > > >
> > > > import java.io.*;
> > > > import java.util.zip.ZipEntry;
> > > > import java.util.zip.ZipFile;
> > > > import org.apache.log.Logger;
> > > > import org.apache.jmeter.threads.JMeterVariables;
> > > >
> > > > public class JmeterZipReader
> > > > {
> > > > private String zipFile;
> > > > private Logger log;
> > > > private String outputVariable;
> > > > private JMeterVariables vars;
> > > >
> > > > JmeterZipReader(String zipFile, Logger log, String
> outputVariable,
> > > > JMeterVariables vars){
> > > > this.zipFile = zipFile;
> > > > this.log = log;
> > > > this.outputVariable = outputVariable;
> > > > this.vars = vars;
> > > > }
> > > >
> > > > // textFile is the name of the text file inside the zip file
> > > > public void read(String textFile){
> > > > ZipFile zip;
> > > > ZipEntry ze;
> > > > InputStream input;
> > > > BufferedReader br;
> > > >
> > > > try {
> > > > zip=new ZipFile(this.zipFile);
> > > > ze=zip.getEntry(textFile);
> > > > input = zip.getInputStream(ze);
> > > > br = new BufferedReader(new InputStreamReader(input,
> > > > "US-ASCII"));
> > > >
> > > > String line;
> > > > int cnt = 0;
> > > > while((line = br.readLine()) != null && cnt <= 10) {
> > > > this.vars.put(this.outputVariable, line);
> > > > log.info(line);
> > > > cnt += 1;
> > > > }
> > > >
> > > > br.close();
> > > > zip.close();
> > > > }
> > > > catch (Exception e) {
> > > > log.error("Unhandled exception:");
> > > > e.printStackTrace();
> > > > }
> > > > }
> > > > }
> > > >
> > > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
> > > > "CONTEXT", ctx.getVariables());
> > > > jzr.read("test.csv");
> > > >
> > > >
> > > >
> > > > Many thanks,
> > > > Janusz
> > > >
> > >
> >
>
Re: How to read a zipped text file and iterate over all the lines
Posted by Deepak Shetty <sh...@gmail.com>.
>that I don't wan't to unzip those files,
The question is why do you want to do that? whats the benefit? - Again you
have setup and tear down threadgroups so you can do it before your test
runs and clean it up after its done (or as part of your build) - note you
have to factor in multiple threads too ..
On Tue, Jul 9, 2013 at 2:28 AM, Janusz Kowalczyk
<ko...@gmail.com>wrote:
> The thing is that I don't wan't to unzip those files, but just read
> directly from them.
>
> OK, maybe easier way would be just returning a list or a map of variables,
> so that I could iterate over it?
>
> ps. I put this strange loop just for the sake of the demo, so it doesn't
> read more that 10 lines at a time :)
>
>
>
> On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
>
> > this.vars.put(this.outputVariable, line); ==> Always overwrites the
> current
> > value
> > should be the last line however unless your loop is also incorrect
> >
> > But the better way is to get your data in the form you want first , then
> > start the test rather than the test writing some code to read the data
> i.e.
> > unzip your files whatever before you mai n test begins (as part of your
> > build or as a startup threadgroup )
> >
> >
> > On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
> > kowalczykjanusz@gmail.com
> > > wrote:
> >
> > > Hi All,
> > >
> > > I'd like to create a script that would read a zipped text file, and for
> > > example for each line (passed as variable) send a HTTP request.
> > >
> > > So far, I've written a script that works partially :)
> > > Partially, because sending a new line to log.info() prints out this
> new
> > > line (which is good :) ), but when I set the value of a variable to
> newly
> > > read line, then the value of that variable remains the same and is
> always
> > > equal to the first line of that text file.
> > >
> > > Please find the code below. To test it:
> > > - create a simple text file with just few lines of random text,
> > > - zip that text file
> > > - Add a JSR223 preprocessor & set the script language to Java
> > > - configure the script accordingly
> > > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
> > > - check the msg value of the http request in "View Results Tree"
> > >
> > >
> > > import java.io.*;
> > > import java.util.zip.ZipEntry;
> > > import java.util.zip.ZipFile;
> > > import org.apache.log.Logger;
> > > import org.apache.jmeter.threads.JMeterVariables;
> > >
> > > public class JmeterZipReader
> > > {
> > > private String zipFile;
> > > private Logger log;
> > > private String outputVariable;
> > > private JMeterVariables vars;
> > >
> > > JmeterZipReader(String zipFile, Logger log, String outputVariable,
> > > JMeterVariables vars){
> > > this.zipFile = zipFile;
> > > this.log = log;
> > > this.outputVariable = outputVariable;
> > > this.vars = vars;
> > > }
> > >
> > > // textFile is the name of the text file inside the zip file
> > > public void read(String textFile){
> > > ZipFile zip;
> > > ZipEntry ze;
> > > InputStream input;
> > > BufferedReader br;
> > >
> > > try {
> > > zip=new ZipFile(this.zipFile);
> > > ze=zip.getEntry(textFile);
> > > input = zip.getInputStream(ze);
> > > br = new BufferedReader(new InputStreamReader(input,
> > > "US-ASCII"));
> > >
> > > String line;
> > > int cnt = 0;
> > > while((line = br.readLine()) != null && cnt <= 10) {
> > > this.vars.put(this.outputVariable, line);
> > > log.info(line);
> > > cnt += 1;
> > > }
> > >
> > > br.close();
> > > zip.close();
> > > }
> > > catch (Exception e) {
> > > log.error("Unhandled exception:");
> > > e.printStackTrace();
> > > }
> > > }
> > > }
> > >
> > > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
> > > "CONTEXT", ctx.getVariables());
> > > jzr.read("test.csv");
> > >
> > >
> > >
> > > Many thanks,
> > > Janusz
> > >
> >
>
Re: How to read a zipped text file and iterate over all the lines
Posted by sebb <se...@gmail.com>.
On 9 July 2013 10:28, Janusz Kowalczyk <ko...@gmail.com> wrote:
> The thing is that I don't wan't to unzip those files, but just read
> directly from them.
In that case, I suggest you look at the function StringFromFile or the
CSV DataSet test element and copy what they do in your own code.
> OK, maybe easier way would be just returning a list or a map of variables,
> so that I could iterate over it?
>
> ps. I put this strange loop just for the sake of the demo, so it doesn't
> read more that 10 lines at a time :)
>
>
>
> On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
>
>> this.vars.put(this.outputVariable, line); ==> Always overwrites the current
>> value
>> should be the last line however unless your loop is also incorrect
>>
>> But the better way is to get your data in the form you want first , then
>> start the test rather than the test writing some code to read the data i.e.
>> unzip your files whatever before you mai n test begins (as part of your
>> build or as a startup threadgroup )
>>
>>
>> On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
>> kowalczykjanusz@gmail.com
>> > wrote:
>>
>> > Hi All,
>> >
>> > I'd like to create a script that would read a zipped text file, and for
>> > example for each line (passed as variable) send a HTTP request.
>> >
>> > So far, I've written a script that works partially :)
>> > Partially, because sending a new line to log.info() prints out this new
>> > line (which is good :) ), but when I set the value of a variable to newly
>> > read line, then the value of that variable remains the same and is always
>> > equal to the first line of that text file.
>> >
>> > Please find the code below. To test it:
>> > - create a simple text file with just few lines of random text,
>> > - zip that text file
>> > - Add a JSR223 preprocessor & set the script language to Java
>> > - configure the script accordingly
>> > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
>> > - check the msg value of the http request in "View Results Tree"
>> >
>> >
>> > import java.io.*;
>> > import java.util.zip.ZipEntry;
>> > import java.util.zip.ZipFile;
>> > import org.apache.log.Logger;
>> > import org.apache.jmeter.threads.JMeterVariables;
>> >
>> > public class JmeterZipReader
>> > {
>> > private String zipFile;
>> > private Logger log;
>> > private String outputVariable;
>> > private JMeterVariables vars;
>> >
>> > JmeterZipReader(String zipFile, Logger log, String outputVariable,
>> > JMeterVariables vars){
>> > this.zipFile = zipFile;
>> > this.log = log;
>> > this.outputVariable = outputVariable;
>> > this.vars = vars;
>> > }
>> >
>> > // textFile is the name of the text file inside the zip file
>> > public void read(String textFile){
>> > ZipFile zip;
>> > ZipEntry ze;
>> > InputStream input;
>> > BufferedReader br;
>> >
>> > try {
>> > zip=new ZipFile(this.zipFile);
>> > ze=zip.getEntry(textFile);
>> > input = zip.getInputStream(ze);
>> > br = new BufferedReader(new InputStreamReader(input,
>> > "US-ASCII"));
>> >
>> > String line;
>> > int cnt = 0;
>> > while((line = br.readLine()) != null && cnt <= 10) {
>> > this.vars.put(this.outputVariable, line);
>> > log.info(line);
>> > cnt += 1;
>> > }
>> >
>> > br.close();
>> > zip.close();
>> > }
>> > catch (Exception e) {
>> > log.error("Unhandled exception:");
>> > e.printStackTrace();
>> > }
>> > }
>> > }
>> >
>> > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
>> > "CONTEXT", ctx.getVariables());
>> > jzr.read("test.csv");
>> >
>> >
>> >
>> > Many thanks,
>> > Janusz
>> >
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@jmeter.apache.org
For additional commands, e-mail: user-help@jmeter.apache.org
Re: How to read a zipped text file and iterate over all the lines
Posted by Janusz Kowalczyk <ko...@gmail.com>.
The thing is that I don't wan't to unzip those files, but just read
directly from them.
OK, maybe easier way would be just returning a list or a map of variables,
so that I could iterate over it?
ps. I put this strange loop just for the sake of the demo, so it doesn't
read more that 10 lines at a time :)
On 8 July 2013 23:35, Deepak Shetty <sh...@gmail.com> wrote:
> this.vars.put(this.outputVariable, line); ==> Always overwrites the current
> value
> should be the last line however unless your loop is also incorrect
>
> But the better way is to get your data in the form you want first , then
> start the test rather than the test writing some code to read the data i.e.
> unzip your files whatever before you mai n test begins (as part of your
> build or as a startup threadgroup )
>
>
> On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <
> kowalczykjanusz@gmail.com
> > wrote:
>
> > Hi All,
> >
> > I'd like to create a script that would read a zipped text file, and for
> > example for each line (passed as variable) send a HTTP request.
> >
> > So far, I've written a script that works partially :)
> > Partially, because sending a new line to log.info() prints out this new
> > line (which is good :) ), but when I set the value of a variable to newly
> > read line, then the value of that variable remains the same and is always
> > equal to the first line of that text file.
> >
> > Please find the code below. To test it:
> > - create a simple text file with just few lines of random text,
> > - zip that text file
> > - Add a JSR223 preprocessor & set the script language to Java
> > - configure the script accordingly
> > - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
> > - check the msg value of the http request in "View Results Tree"
> >
> >
> > import java.io.*;
> > import java.util.zip.ZipEntry;
> > import java.util.zip.ZipFile;
> > import org.apache.log.Logger;
> > import org.apache.jmeter.threads.JMeterVariables;
> >
> > public class JmeterZipReader
> > {
> > private String zipFile;
> > private Logger log;
> > private String outputVariable;
> > private JMeterVariables vars;
> >
> > JmeterZipReader(String zipFile, Logger log, String outputVariable,
> > JMeterVariables vars){
> > this.zipFile = zipFile;
> > this.log = log;
> > this.outputVariable = outputVariable;
> > this.vars = vars;
> > }
> >
> > // textFile is the name of the text file inside the zip file
> > public void read(String textFile){
> > ZipFile zip;
> > ZipEntry ze;
> > InputStream input;
> > BufferedReader br;
> >
> > try {
> > zip=new ZipFile(this.zipFile);
> > ze=zip.getEntry(textFile);
> > input = zip.getInputStream(ze);
> > br = new BufferedReader(new InputStreamReader(input,
> > "US-ASCII"));
> >
> > String line;
> > int cnt = 0;
> > while((line = br.readLine()) != null && cnt <= 10) {
> > this.vars.put(this.outputVariable, line);
> > log.info(line);
> > cnt += 1;
> > }
> >
> > br.close();
> > zip.close();
> > }
> > catch (Exception e) {
> > log.error("Unhandled exception:");
> > e.printStackTrace();
> > }
> > }
> > }
> >
> > JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
> > "CONTEXT", ctx.getVariables());
> > jzr.read("test.csv");
> >
> >
> >
> > Many thanks,
> > Janusz
> >
>
Re: How to read a zipped text file and iterate over all the lines
Posted by Deepak Shetty <sh...@gmail.com>.
this.vars.put(this.outputVariable, line); ==> Always overwrites the current
value
should be the last line however unless your loop is also incorrect
But the better way is to get your data in the form you want first , then
start the test rather than the test writing some code to read the data i.e.
unzip your files whatever before you mai n test begins (as part of your
build or as a startup threadgroup )
On Mon, Jul 8, 2013 at 12:42 PM, Janusz Kowalczyk <kowalczykjanusz@gmail.com
> wrote:
> Hi All,
>
> I'd like to create a script that would read a zipped text file, and for
> example for each line (passed as variable) send a HTTP request.
>
> So far, I've written a script that works partially :)
> Partially, because sending a new line to log.info() prints out this new
> line (which is good :) ), but when I set the value of a variable to newly
> read line, then the value of that variable remains the same and is always
> equal to the first line of that text file.
>
> Please find the code below. To test it:
> - create a simple text file with just few lines of random text,
> - zip that text file
> - Add a JSR223 preprocessor & set the script language to Java
> - configure the script accordingly
> - add a HTTP sampler and add ${CONTEXT} to the "RAW Post Body" field
> - check the msg value of the http request in "View Results Tree"
>
>
> import java.io.*;
> import java.util.zip.ZipEntry;
> import java.util.zip.ZipFile;
> import org.apache.log.Logger;
> import org.apache.jmeter.threads.JMeterVariables;
>
> public class JmeterZipReader
> {
> private String zipFile;
> private Logger log;
> private String outputVariable;
> private JMeterVariables vars;
>
> JmeterZipReader(String zipFile, Logger log, String outputVariable,
> JMeterVariables vars){
> this.zipFile = zipFile;
> this.log = log;
> this.outputVariable = outputVariable;
> this.vars = vars;
> }
>
> // textFile is the name of the text file inside the zip file
> public void read(String textFile){
> ZipFile zip;
> ZipEntry ze;
> InputStream input;
> BufferedReader br;
>
> try {
> zip=new ZipFile(this.zipFile);
> ze=zip.getEntry(textFile);
> input = zip.getInputStream(ze);
> br = new BufferedReader(new InputStreamReader(input,
> "US-ASCII"));
>
> String line;
> int cnt = 0;
> while((line = br.readLine()) != null && cnt <= 10) {
> this.vars.put(this.outputVariable, line);
> log.info(line);
> cnt += 1;
> }
>
> br.close();
> zip.close();
> }
> catch (Exception e) {
> log.error("Unhandled exception:");
> e.printStackTrace();
> }
> }
> }
>
> JmeterZipReader jzr = new JmeterZipReader("/path/to/a/test.zip", log,
> "CONTEXT", ctx.getVariables());
> jzr.read("test.csv");
>
>
>
> Many thanks,
> Janusz
>