You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kirill Kozlov <ki...@google.com> on 2019/11/07 00:44:28 UTC
(Question) SQL integration tests for MongoDb
Hi everyone!
I am trying to test MongoDb Sql Table, but not quite sure how to pass
pipeline options with the hostName, port, and databaseName used by Jenkins.
It looks like the integration test for MongoDbIO Connector obtain those
values from the
'beam/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy' file
via calling the following methods in the 'gradle.build' file:
provideIntegrationTestingDependencies()
enableJavaPerformanceTesting()
Sql build file already has a task with the name 'integrationTest' defined
and does not let us do `enableJavaPerformanceTesting()`.
I would really appreciate if someone could provide me with a couple of
pointers on getting this to work.
-
Kirill
Re: (Question) SQL integration tests for MongoDb
Posted by Kirill Kozlov <ki...@google.com>.
Alternative approach would be to manually start a MongoDb service like it
is done here:
https://github.com/apache/beam/blob/master/sdks/java/io/mongodb/src/test/java/org/apache/beam/sdk/io/mongodb/MongoDbIOTest.java#L85
Doing it like in the example above should solve my problem.
Thank you for your help!
-
Kirill
On Fri, Nov 8, 2019, 03:09 Michał Walenia <mi...@polidea.com>
wrote:
> Won't the command be analogous to what is in the Javadoc of
> MongoDbReadWriteIT? It seems that you don't need to use
> `enableJavaPerformanceTesting`, as `integrationTest` task parses
> `pipelineOptions` parameter.
>
>
>
> On Thu, Nov 7, 2019 at 6:40 PM Kirill Kozlov <ki...@google.com>
> wrote:
>
>> Thank you for your response!
>>
>> I want to make sure that when tests run on Jenkins they get supplied with
>> pipelines options containing hostName and Port of a running MongoDb service.
>>
>> I'm writing integration test for a MongoDb SQL adapter (located
>> sdks/java/extensions/sql/meta/provider/mongodb).
>> I cannot simply use `enableJavaPerformanceTesting()`, because tests for
>> all adapters are run via the same build file, which has a custom task
>> "integrationTest".
>>
>> I hope this better explains the problem I am trying to tackle.
>>
>> -
>> Kirill
>>
>> On Thu, Nov 7, 2019, 03:36 Michał Walenia <mi...@polidea.com>
>> wrote:
>>
>>> Hi,
>>>
>>> What exactly are you trying to do? If you're looking for a way to
>>> provide pipeline options to the MongoDBIOIT, you can pass them via command
>>> line like this:
>>>
>>> ./gradlew integrationTest -p sdks/java/io/mongodb
>>>
>>>
>>>
>>> * -DintegrationTestPipelineOptions='[ "--mongoDBHostName=1.2.3.4",
>>> "--mongoDBPort=27017", "--mongoDBDatabaseName=mypass",
>>> "--numberOfRecords=1000" ]'*
>>> --tests org.apache.beam.sdk.io.mongodb.MongoDbIOIT
>>> -DintegrationTestRunner=direct
>>>
>>> Gradle tasks created with `enableJavaPerformanceTesting()` will allow
>>> such options to be passed.
>>>
>>> If you're trying to do something else, please let me know.
>>>
>>> Regards
>>> Michal
>>>
>>> On Thu, Nov 7, 2019 at 1:44 AM Kirill Kozlov <ki...@google.com>
>>> wrote:
>>>
>>>> Hi everyone!
>>>>
>>>> I am trying to test MongoDb Sql Table, but not quite sure how to pass
>>>> pipeline options with the hostName, port, and databaseName used by Jenkins.
>>>>
>>>> It looks like the integration test for MongoDbIO Connector obtain those
>>>> values from the
>>>> 'beam/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy' file
>>>> via calling the following methods in the 'gradle.build' file:
>>>> provideIntegrationTestingDependencies()
>>>> enableJavaPerformanceTesting()
>>>>
>>>> Sql build file already has a task with the name 'integrationTest'
>>>> defined and does not let us do `enableJavaPerformanceTesting()`.
>>>>
>>>> I would really appreciate if someone could provide me with a couple of
>>>> pointers on getting this to work.
>>>>
>>>> -
>>>> Kirill
>>>>
>>>
>>>
>>> --
>>>
>>> Michał Walenia
>>> Polidea <https://www.polidea.com/> | Software Engineer
>>>
>>> M: +48 791 432 002 <+48791432002>
>>> E: michal.walenia@polidea.com
>>>
>>> Unique Tech
>>> Check out our projects! <https://www.polidea.com/our-work>
>>>
>>
>
> --
>
> Michał Walenia
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.walenia@polidea.com
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
Re: (Question) SQL integration tests for MongoDb
Posted by Michał Walenia <mi...@polidea.com>.
Won't the command be analogous to what is in the Javadoc of
MongoDbReadWriteIT? It seems that you don't need to use
`enableJavaPerformanceTesting`, as `integrationTest` task parses
`pipelineOptions` parameter.
On Thu, Nov 7, 2019 at 6:40 PM Kirill Kozlov <ki...@google.com>
wrote:
> Thank you for your response!
>
> I want to make sure that when tests run on Jenkins they get supplied with
> pipelines options containing hostName and Port of a running MongoDb service.
>
> I'm writing integration test for a MongoDb SQL adapter (located
> sdks/java/extensions/sql/meta/provider/mongodb).
> I cannot simply use `enableJavaPerformanceTesting()`, because tests for
> all adapters are run via the same build file, which has a custom task
> "integrationTest".
>
> I hope this better explains the problem I am trying to tackle.
>
> -
> Kirill
>
> On Thu, Nov 7, 2019, 03:36 Michał Walenia <mi...@polidea.com>
> wrote:
>
>> Hi,
>>
>> What exactly are you trying to do? If you're looking for a way to provide
>> pipeline options to the MongoDBIOIT, you can pass them via command line
>> like this:
>>
>> ./gradlew integrationTest -p sdks/java/io/mongodb
>>
>>
>>
>> * -DintegrationTestPipelineOptions='[ "--mongoDBHostName=1.2.3.4",
>> "--mongoDBPort=27017", "--mongoDBDatabaseName=mypass",
>> "--numberOfRecords=1000" ]'*
>> --tests org.apache.beam.sdk.io.mongodb.MongoDbIOIT
>> -DintegrationTestRunner=direct
>>
>> Gradle tasks created with `enableJavaPerformanceTesting()` will allow
>> such options to be passed.
>>
>> If you're trying to do something else, please let me know.
>>
>> Regards
>> Michal
>>
>> On Thu, Nov 7, 2019 at 1:44 AM Kirill Kozlov <ki...@google.com>
>> wrote:
>>
>>> Hi everyone!
>>>
>>> I am trying to test MongoDb Sql Table, but not quite sure how to pass
>>> pipeline options with the hostName, port, and databaseName used by Jenkins.
>>>
>>> It looks like the integration test for MongoDbIO Connector obtain those
>>> values from the
>>> 'beam/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy' file
>>> via calling the following methods in the 'gradle.build' file:
>>> provideIntegrationTestingDependencies()
>>> enableJavaPerformanceTesting()
>>>
>>> Sql build file already has a task with the name 'integrationTest'
>>> defined and does not let us do `enableJavaPerformanceTesting()`.
>>>
>>> I would really appreciate if someone could provide me with a couple of
>>> pointers on getting this to work.
>>>
>>> -
>>> Kirill
>>>
>>
>>
>> --
>>
>> Michał Walenia
>> Polidea <https://www.polidea.com/> | Software Engineer
>>
>> M: +48 791 432 002 <+48791432002>
>> E: michal.walenia@polidea.com
>>
>> Unique Tech
>> Check out our projects! <https://www.polidea.com/our-work>
>>
>
--
Michał Walenia
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 791 432 002 <+48791432002>
E: michal.walenia@polidea.com
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>
Re: (Question) SQL integration tests for MongoDb
Posted by Kirill Kozlov <ki...@google.com>.
Thank you for your response!
I want to make sure that when tests run on Jenkins they get supplied with
pipelines options containing hostName and Port of a running MongoDb service.
I'm writing integration test for a MongoDb SQL adapter (located
sdks/java/extensions/sql/meta/provider/mongodb).
I cannot simply use `enableJavaPerformanceTesting()`, because tests for all
adapters are run via the same build file, which has a custom task
"integrationTest".
I hope this better explains the problem I am trying to tackle.
-
Kirill
On Thu, Nov 7, 2019, 03:36 Michał Walenia <mi...@polidea.com>
wrote:
> Hi,
>
> What exactly are you trying to do? If you're looking for a way to provide
> pipeline options to the MongoDBIOIT, you can pass them via command line
> like this:
>
> ./gradlew integrationTest -p sdks/java/io/mongodb
>
>
>
> * -DintegrationTestPipelineOptions='[ "--mongoDBHostName=1.2.3.4",
> "--mongoDBPort=27017", "--mongoDBDatabaseName=mypass",
> "--numberOfRecords=1000" ]'*
> --tests org.apache.beam.sdk.io.mongodb.MongoDbIOIT
> -DintegrationTestRunner=direct
>
> Gradle tasks created with `enableJavaPerformanceTesting()` will allow such
> options to be passed.
>
> If you're trying to do something else, please let me know.
>
> Regards
> Michal
>
> On Thu, Nov 7, 2019 at 1:44 AM Kirill Kozlov <ki...@google.com>
> wrote:
>
>> Hi everyone!
>>
>> I am trying to test MongoDb Sql Table, but not quite sure how to pass
>> pipeline options with the hostName, port, and databaseName used by Jenkins.
>>
>> It looks like the integration test for MongoDbIO Connector obtain those
>> values from the
>> 'beam/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy' file
>> via calling the following methods in the 'gradle.build' file:
>> provideIntegrationTestingDependencies()
>> enableJavaPerformanceTesting()
>>
>> Sql build file already has a task with the name 'integrationTest' defined
>> and does not let us do `enableJavaPerformanceTesting()`.
>>
>> I would really appreciate if someone could provide me with a couple of
>> pointers on getting this to work.
>>
>> -
>> Kirill
>>
>
>
> --
>
> Michał Walenia
> Polidea <https://www.polidea.com/> | Software Engineer
>
> M: +48 791 432 002 <+48791432002>
> E: michal.walenia@polidea.com
>
> Unique Tech
> Check out our projects! <https://www.polidea.com/our-work>
>
Re: (Question) SQL integration tests for MongoDb
Posted by Michał Walenia <mi...@polidea.com>.
Hi,
What exactly are you trying to do? If you're looking for a way to provide
pipeline options to the MongoDBIOIT, you can pass them via command line
like this:
./gradlew integrationTest -p sdks/java/io/mongodb
* -DintegrationTestPipelineOptions='[ "--mongoDBHostName=1.2.3.4",
"--mongoDBPort=27017", "--mongoDBDatabaseName=mypass",
"--numberOfRecords=1000" ]'*
--tests org.apache.beam.sdk.io.mongodb.MongoDbIOIT
-DintegrationTestRunner=direct
Gradle tasks created with `enableJavaPerformanceTesting()` will allow such
options to be passed.
If you're trying to do something else, please let me know.
Regards
Michal
On Thu, Nov 7, 2019 at 1:44 AM Kirill Kozlov <ki...@google.com>
wrote:
> Hi everyone!
>
> I am trying to test MongoDb Sql Table, but not quite sure how to pass
> pipeline options with the hostName, port, and databaseName used by Jenkins.
>
> It looks like the integration test for MongoDbIO Connector obtain those
> values from the
> 'beam/.test-infra/jenkins/job_PerformanceTests_MongoDBIO_IT.groovy' file
> via calling the following methods in the 'gradle.build' file:
> provideIntegrationTestingDependencies()
> enableJavaPerformanceTesting()
>
> Sql build file already has a task with the name 'integrationTest' defined
> and does not let us do `enableJavaPerformanceTesting()`.
>
> I would really appreciate if someone could provide me with a couple of
> pointers on getting this to work.
>
> -
> Kirill
>
--
Michał Walenia
Polidea <https://www.polidea.com/> | Software Engineer
M: +48 791 432 002 <+48791432002>
E: michal.walenia@polidea.com
Unique Tech
Check out our projects! <https://www.polidea.com/our-work>