You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Faisal Maqsood <fa...@venturedive.com> on 2020/12/16 10:54:29 UTC
Query Related with Load Test Running on Local Machine.
Hey everyone,
I need some help related to running LoadTest on the local machine.
Reference to the file (*apache_beam.io.gcp.bigquery_read_perf_test* ) I
find the command to run LoadTest but I am not sure about the parameters to
be passed in this command can someone please help me out on this and give
any example command.
what value should be passed in these parameters
*metrics_dataset = ...metrics_table* *= ...*
*staging_location = ...*
*temp_location = ...*
Example test run on DataflowRunner:
python -m apache_beam.io.gcp.bigquery_read_perf_test \
--test-pipeline-options="
--runner=TestDataflowRunner
--project=...
--region=...
--staging_location=gs://...
--temp_location=gs://...
--sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
--publish_to_big_query=true
--metrics_dataset=gs://...
--metrics_table=...
--input_dataset=...
--input_table=...
--input_options='{
\"num_records\": 1024,
\"key_size\": 1,
\"value_size\": 1024,
}
Thanks in advance :)
*Faisal Ali*
Senior Software Engineer
Mobile: +92 3433016854
Skype: fessimax
<http://venturedive.com/>
Re: Query Related with Load Test Running on Local Machine.
Posted by Pablo Estrada <pa...@google.com>.
You can simply pass --publish_to_big_query=false - this flag is used to
publish test results to BigQuery, for us to keep a history of the
performance of the tests.
If you want that, then provide a BQ dataset for your performance history,
but if you only need to run the test once and see the results, you don't
need to pass any GCP-related info.
Best
-P.
On Mon, Dec 21, 2020 at 10:57 AM Ahmet Altay <al...@google.com> wrote:
> In case this helps there is an example of running tests on Dataflow here
> [1]. Runner related information needs to be changed for running locally.
> Does this help?
>
> Faisal, what is the purpose for running the load test locally (assuming
> you mean direct runner)? Is this related to BEAM-10029?
>
> [1]
> https://github.com/apache/beam/blob/164a7b68a477978d869c6a2bf3faf80135373390/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy#L26
>
> On Wed, Dec 16, 2020 at 11:13 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> I haven't tried this, but in general.
>>
>> metrics_dataset = A BigQuery dataset (to publish metrics)
>> metrics_table = A BigQuery table name
>> staging_location = A GCS bucket (of the form gs://...) (also I think this
>> should be optional since 'staging_location' defaults to 'temp_location')
>> temp_location = A GCS bucket (of the form gs://...)
>>
>> Also, note that this command assumes a DataflowRunner, hence you'll have
>> to perform Dataflow specific setup mentioned here:
>> https://beam.apache.org/documentation/runners/dataflow/#setup
>>
>> Thanks,
>> Cham
>>
>>
>> On Wed, Dec 16, 2020 at 10:51 AM Ahmet Altay <al...@google.com> wrote:
>>
>>> +Pablo Estrada <pa...@google.com> +Heejong Lee <he...@google.com> +Chamikara
>>> Jayalath <ch...@google.com>
>>>
>>> On Wed, Dec 16, 2020 at 9:15 AM Faisal Maqsood <
>>> faisal.maqsood@venturedive.com> wrote:
>>>
>>>> Hey everyone,
>>>> I need some help related to running LoadTest on the local machine.
>>>> Reference to the file (*apache_beam.io.gcp.bigquery_read_perf_test* )
>>>> I find the command to run LoadTest but I am not sure about the parameters
>>>> to be passed in this command can someone please help me out on this and
>>>> give any example command.
>>>>
>>>> what value should be passed in these parameters
>>>>
>>>> *metrics_dataset = ...metrics_table* *= ...*
>>>>
>>>> *staging_location = ...*
>>>>
>>>> *temp_location = ...*
>>>>
>>>> Example test run on DataflowRunner:
>>>>
>>>> python -m apache_beam.io.gcp.bigquery_read_perf_test \
>>>> --test-pipeline-options="
>>>> --runner=TestDataflowRunner
>>>> --project=...
>>>> --region=...
>>>> --staging_location=gs://...
>>>> --temp_location=gs://...
>>>> --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
>>>> --publish_to_big_query=true
>>>> --metrics_dataset=gs://...
>>>> --metrics_table=...
>>>> --input_dataset=...
>>>> --input_table=...
>>>> --input_options='{
>>>> \"num_records\": 1024,
>>>> \"key_size\": 1,
>>>> \"value_size\": 1024,
>>>> }
>>>>
>>>> Thanks in advance :)
>>>>
>>>> *Faisal Ali*
>>>> Senior Software Engineer
>>>> Mobile: +92 3433016854 <+92%20343%203016854>
>>>> Skype: fessimax
>>>>
>>>> <http://venturedive.com/>
>>>>
>>>
Re: Query Related with Load Test Running on Local Machine.
Posted by Ahmet Altay <al...@google.com>.
In case this helps there is an example of running tests on Dataflow here
[1]. Runner related information needs to be changed for running locally.
Does this help?
Faisal, what is the purpose for running the load test locally (assuming you
mean direct runner)? Is this related to BEAM-10029?
[1]
https://github.com/apache/beam/blob/164a7b68a477978d869c6a2bf3faf80135373390/.test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy#L26
On Wed, Dec 16, 2020 at 11:13 AM Chamikara Jayalath <ch...@google.com>
wrote:
> I haven't tried this, but in general.
>
> metrics_dataset = A BigQuery dataset (to publish metrics)
> metrics_table = A BigQuery table name
> staging_location = A GCS bucket (of the form gs://...) (also I think this
> should be optional since 'staging_location' defaults to 'temp_location')
> temp_location = A GCS bucket (of the form gs://...)
>
> Also, note that this command assumes a DataflowRunner, hence you'll have
> to perform Dataflow specific setup mentioned here:
> https://beam.apache.org/documentation/runners/dataflow/#setup
>
> Thanks,
> Cham
>
>
> On Wed, Dec 16, 2020 at 10:51 AM Ahmet Altay <al...@google.com> wrote:
>
>> +Pablo Estrada <pa...@google.com> +Heejong Lee <he...@google.com> +Chamikara
>> Jayalath <ch...@google.com>
>>
>> On Wed, Dec 16, 2020 at 9:15 AM Faisal Maqsood <
>> faisal.maqsood@venturedive.com> wrote:
>>
>>> Hey everyone,
>>> I need some help related to running LoadTest on the local machine.
>>> Reference to the file (*apache_beam.io.gcp.bigquery_read_perf_test* )
>>> I find the command to run LoadTest but I am not sure about the parameters
>>> to be passed in this command can someone please help me out on this and
>>> give any example command.
>>>
>>> what value should be passed in these parameters
>>>
>>> *metrics_dataset = ...metrics_table* *= ...*
>>>
>>> *staging_location = ...*
>>>
>>> *temp_location = ...*
>>>
>>> Example test run on DataflowRunner:
>>>
>>> python -m apache_beam.io.gcp.bigquery_read_perf_test \
>>> --test-pipeline-options="
>>> --runner=TestDataflowRunner
>>> --project=...
>>> --region=...
>>> --staging_location=gs://...
>>> --temp_location=gs://...
>>> --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
>>> --publish_to_big_query=true
>>> --metrics_dataset=gs://...
>>> --metrics_table=...
>>> --input_dataset=...
>>> --input_table=...
>>> --input_options='{
>>> \"num_records\": 1024,
>>> \"key_size\": 1,
>>> \"value_size\": 1024,
>>> }
>>>
>>> Thanks in advance :)
>>>
>>> *Faisal Ali*
>>> Senior Software Engineer
>>> Mobile: +92 3433016854 <+92%20343%203016854>
>>> Skype: fessimax
>>>
>>> <http://venturedive.com/>
>>>
>>
Re: Query Related with Load Test Running on Local Machine.
Posted by Chamikara Jayalath <ch...@google.com>.
I haven't tried this, but in general.
metrics_dataset = A BigQuery dataset (to publish metrics)
metrics_table = A BigQuery table name
staging_location = A GCS bucket (of the form gs://...) (also I think this
should be optional since 'staging_location' defaults to 'temp_location')
temp_location = A GCS bucket (of the form gs://...)
Also, note that this command assumes a DataflowRunner, hence you'll have to
perform Dataflow specific setup mentioned here:
https://beam.apache.org/documentation/runners/dataflow/#setup
Thanks,
Cham
On Wed, Dec 16, 2020 at 10:51 AM Ahmet Altay <al...@google.com> wrote:
> +Pablo Estrada <pa...@google.com> +Heejong Lee <he...@google.com> +Chamikara
> Jayalath <ch...@google.com>
>
> On Wed, Dec 16, 2020 at 9:15 AM Faisal Maqsood <
> faisal.maqsood@venturedive.com> wrote:
>
>> Hey everyone,
>> I need some help related to running LoadTest on the local machine.
>> Reference to the file (*apache_beam.io.gcp.bigquery_read_perf_test* ) I
>> find the command to run LoadTest but I am not sure about the parameters to
>> be passed in this command can someone please help me out on this and give
>> any example command.
>>
>> what value should be passed in these parameters
>>
>> *metrics_dataset = ...metrics_table* *= ...*
>>
>> *staging_location = ...*
>>
>> *temp_location = ...*
>>
>> Example test run on DataflowRunner:
>>
>> python -m apache_beam.io.gcp.bigquery_read_perf_test \
>> --test-pipeline-options="
>> --runner=TestDataflowRunner
>> --project=...
>> --region=...
>> --staging_location=gs://...
>> --temp_location=gs://...
>> --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
>> --publish_to_big_query=true
>> --metrics_dataset=gs://...
>> --metrics_table=...
>> --input_dataset=...
>> --input_table=...
>> --input_options='{
>> \"num_records\": 1024,
>> \"key_size\": 1,
>> \"value_size\": 1024,
>> }
>>
>> Thanks in advance :)
>>
>> *Faisal Ali*
>> Senior Software Engineer
>> Mobile: +92 3433016854 <+92%20343%203016854>
>> Skype: fessimax
>>
>> <http://venturedive.com/>
>>
>
Re: Query Related with Load Test Running on Local Machine.
Posted by Ahmet Altay <al...@google.com>.
+Pablo Estrada <pa...@google.com> +Heejong Lee <he...@google.com>
+Chamikara
Jayalath <ch...@google.com>
On Wed, Dec 16, 2020 at 9:15 AM Faisal Maqsood <
faisal.maqsood@venturedive.com> wrote:
> Hey everyone,
> I need some help related to running LoadTest on the local machine.
> Reference to the file (*apache_beam.io.gcp.bigquery_read_perf_test* ) I
> find the command to run LoadTest but I am not sure about the parameters to
> be passed in this command can someone please help me out on this and give
> any example command.
>
> what value should be passed in these parameters
>
> *metrics_dataset = ...metrics_table* *= ...*
>
> *staging_location = ...*
>
> *temp_location = ...*
>
> Example test run on DataflowRunner:
>
> python -m apache_beam.io.gcp.bigquery_read_perf_test \
> --test-pipeline-options="
> --runner=TestDataflowRunner
> --project=...
> --region=...
> --staging_location=gs://...
> --temp_location=gs://...
> --sdk_location=.../dist/apache-beam-x.x.x.dev0.tar.gz
> --publish_to_big_query=true
> --metrics_dataset=gs://...
> --metrics_table=...
> --input_dataset=...
> --input_table=...
> --input_options='{
> \"num_records\": 1024,
> \"key_size\": 1,
> \"value_size\": 1024,
> }
>
> Thanks in advance :)
>
> *Faisal Ali*
> Senior Software Engineer
> Mobile: +92 3433016854 <+92%20343%203016854>
> Skype: fessimax
>
> <http://venturedive.com/>
>