You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2021/07/28 19:20:45 UTC
Command line to run DatstoreIO integration tests for java
Is it possible to run a Datastore IO integration test to test this PR?
https://github.com/apache/beam/pull/15183/files
Probably this test can be ran somehow. Though I don't know the gradle
command to run it
https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
Does anyone know how to run this test?
Re: Command line to run DatstoreIO integration tests for java
Posted by Kyle Weaver <kc...@google.com>.
HI Miguel, you'll need to write a query that parses correctly, but hits a
runtime error on Datastore. Another possibility is you could mock the
Datastore table to throw the error you want. Is there a jira ticket
describing the problem you're trying to solve?
On Fri, Aug 20, 2021 at 3:11 PM Miguel Anzo Palomo <mi...@wizeline.com>
wrote:
> Hi, thanks for the help. I'm able to run the test locally by running the
> sql postCommit :sdks:java:extensions:sql:postCommit. However my problem
> comes when trying to change something in the query in order to inject an
> error, for example trying to change the table name in a query in order to
> test the error path. The problem is that in the integration tests when
> making the call to the datastore read and write transforms the query is
> provided by creating a BeamRelNode created with the query
> sqlEnv.parseQuery(query)
> <https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105>
> which will throw a ParseException if I try to use a query with a non
> existent table name. Is there a way to execute the integration test with a
> query that will cause an error?
>
>
> On Wed, Aug 18, 2021 at 1:28 PM Kyle Weaver <kc...@google.com> wrote:
>
>> There's a hard-coded "include" clause in the source. You can change that.
>> https://github.com/apache/beam/blob/a0fbe00ef12b72ec89672ab32ccc6d6331ca5edd/sdks/java/extensions/sql/build.gradle#L198
>>
>> On Wed, Aug 18, 2021 at 11:19 AM Andrew Pilloud <ap...@google.com>
>> wrote:
>>
>>> You might be able to use the '--tests' flag. (This works for unit
>>> tests.) For example:
>>>
>>> ./gradlew :sdks:java:extensions:sql:postCommit
>>> --tests org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead
>>>
>>> On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath <
>>> chamikara@google.com> wrote:
>>>
>>>> I don't have the Gradle command to run an individual IT unfortunately.
>>>> I usually invoke the corresponding Jenkins test suite (PostCommit SQL in
>>>> this case).
>>>> Note that there are additional Datastore read/write IT tests that do
>>>> not involve SQL (and run as a part of Java PostCommit).
>>>>
>>>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
>>>>
>>>> Thanks,
>>>> Cham
>>>>
>>>> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> +Chamikara, +Heejong,
>>>>>
>>>>> Do you have any ideas here? I want to verify the library does indeed
>>>>> produce exceptions as we expect and that we can catch them as done here:
>>>>> https://github.com/apache/beam/pull/15183
>>>>>
>>>>> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
>>>>> miguel.anzo@wizeline.com> wrote:
>>>>>
>>>>>> Hi, I was able to run the integration test locally, but I'm having
>>>>>> some troubles injecting an error to inject an error here [1] to trigger a
>>>>>> DatastoreException here [2]. The problem is that if I try to change the
>>>>>> table or fields in the select query, an exception is raised in [1] when
>>>>>> parsing the sql to create the BeamRelNode. Would anyone know a way to
>>>>>> inject an error in the integration test to cause a DatastoreException in
>>>>>> [2]?
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
>>>>>> [2]
>>>>>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>>>>>>
>>>>>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
>>>>>> miguel.anzo@wizeline.com> wrote:
>>>>>>
>>>>>>> Hi Alex,
>>>>>>> My problem trying to locally run that test was that, since I don't
>>>>>>> have access to the apache-beam-testing GCP project I was having problems
>>>>>>> getting a datastore instance to read/write from.
>>>>>>>
>>>>>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Miguel, I believe you were having trouble running this test, you
>>>>>>>> mentioned that you needed a Datastore instance.
>>>>>>>> Would you mind sharing the errors or any issue you had when you
>>>>>>>> tried Ke's command?
>>>>>>>>
>>>>>>>> Ke, Do we need to specify a datastore instance to run this test?
>>>>>>>> Where will it read/write from? I assume it will it default to something in
>>>>>>>> the apache-beam-testing GCP project, as other tests do.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> “Run SQL PostCommit” is essentially running
>>>>>>>>> “./gradlew :sqlPostCommit” [1]
>>>>>>>>>
>>>>>>>>> In your case where you would like to run DataStoreReadWriteIT
>>>>>>>>> only, it can be simplified [2] to
>>>>>>>>>
>>>>>>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Ke
>>>>>>>>>
>>>>>>>>> [1]
>>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>>>>>>
>>>>>>>>> [2]
>>>>>>>>> https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I was hoping for the command line to run it. So that the test
>>>>>>>>> could be tweaked to inject an error, and ensure the error handling code
>>>>>>>>> works as expected
>>>>>>>>>
>>>>>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post
>>>>>>>>>> commit integration tests for SQL, which I suppose includes
>>>>>>>>>> DataStoreReadWriteIT
>>>>>>>>>>
>>>>>>>>>> Let me know if whether or not this is sufficient.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>> Ke
>>>>>>>>>>
>>>>>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Is it possible to run a Datastore IO integration test to test
>>>>>>>>>> this PR?
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>>>>>>
>>>>>>>>>> Probably this test can be ran somehow. Though I don't know the
>>>>>>>>>> gradle command to run it
>>>>>>>>>>
>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>>>>>>
>>>>>>>>>> Does anyone know how to run this test?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>>>>
>>>>>>> Software Engineer
>>>>>>>
>>>>>>> miguel.anzo@wizeline.com
>>>>>>>
>>>>>>> Remote Office
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>>>
>>>>>> Software Engineer
>>>>>>
>>>>>> miguel.anzo@wizeline.com
>>>>>>
>>>>>> Remote Office
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *This email and its contents (including any attachments) are being
>>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>>> norepresentation is made with respect to any content contained in this
>>>>>> email.*
>>>>>
>>>>>
>
> --
>
> Miguel Angel Anzo Palomo | WIZELINE
>
> Software Engineer
>
> miguel.anzo@wizeline.com
>
> Remote Office
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
Re: Command line to run DatstoreIO integration tests for java
Posted by Miguel Anzo Palomo <mi...@wizeline.com>.
Hi, thanks for the help. I'm able to run the test locally by running the
sql postCommit :sdks:java:extensions:sql:postCommit. However my problem
comes when trying to change something in the query in order to inject an
error, for example trying to change the table name in a query in order to
test the error path. The problem is that in the integration tests when
making the call to the datastore read and write transforms the query is
provided by creating a BeamRelNode created with the query
sqlEnv.parseQuery(query)
<https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105>
which will throw a ParseException if I try to use a query with a non
existent table name. Is there a way to execute the integration test with a
query that will cause an error?
On Wed, Aug 18, 2021 at 1:28 PM Kyle Weaver <kc...@google.com> wrote:
> There's a hard-coded "include" clause in the source. You can change that.
> https://github.com/apache/beam/blob/a0fbe00ef12b72ec89672ab32ccc6d6331ca5edd/sdks/java/extensions/sql/build.gradle#L198
>
> On Wed, Aug 18, 2021 at 11:19 AM Andrew Pilloud <ap...@google.com>
> wrote:
>
>> You might be able to use the '--tests' flag. (This works for unit tests.)
>> For example:
>>
>> ./gradlew :sdks:java:extensions:sql:postCommit
>> --tests org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead
>>
>> On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath <ch...@google.com>
>> wrote:
>>
>>> I don't have the Gradle command to run an individual IT unfortunately. I
>>> usually invoke the corresponding Jenkins test suite (PostCommit SQL in this
>>> case).
>>> Note that there are additional Datastore read/write IT tests that do not
>>> involve SQL (and run as a part of Java PostCommit).
>>>
>>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
>>>
>>> Thanks,
>>> Cham
>>>
>>> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>> +Chamikara, +Heejong,
>>>>
>>>> Do you have any ideas here? I want to verify the library does indeed
>>>> produce exceptions as we expect and that we can catch them as done here:
>>>> https://github.com/apache/beam/pull/15183
>>>>
>>>> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
>>>> miguel.anzo@wizeline.com> wrote:
>>>>
>>>>> Hi, I was able to run the integration test locally, but I'm having
>>>>> some troubles injecting an error to inject an error here [1] to trigger a
>>>>> DatastoreException here [2]. The problem is that if I try to change the
>>>>> table or fields in the select query, an exception is raised in [1] when
>>>>> parsing the sql to create the BeamRelNode. Would anyone know a way to
>>>>> inject an error in the integration test to cause a DatastoreException in
>>>>> [2]?
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
>>>>> [2]
>>>>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>>>>>
>>>>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
>>>>> miguel.anzo@wizeline.com> wrote:
>>>>>
>>>>>> Hi Alex,
>>>>>> My problem trying to locally run that test was that, since I don't
>>>>>> have access to the apache-beam-testing GCP project I was having problems
>>>>>> getting a datastore instance to read/write from.
>>>>>>
>>>>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Miguel, I believe you were having trouble running this test, you
>>>>>>> mentioned that you needed a Datastore instance.
>>>>>>> Would you mind sharing the errors or any issue you had when you
>>>>>>> tried Ke's command?
>>>>>>>
>>>>>>> Ke, Do we need to specify a datastore instance to run this test?
>>>>>>> Where will it read/write from? I assume it will it default to something in
>>>>>>> the apache-beam-testing GCP project, as other tests do.
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>>
>>>>>>>> “Run SQL PostCommit” is essentially running
>>>>>>>> “./gradlew :sqlPostCommit” [1]
>>>>>>>>
>>>>>>>> In your case where you would like to run DataStoreReadWriteIT only,
>>>>>>>> it can be simplified [2] to
>>>>>>>>
>>>>>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Ke
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>>>>>
>>>>>>>> [2]
>>>>>>>> https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>>>>>
>>>>>>>>
>>>>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>>>>>>
>>>>>>>> I was hoping for the command line to run it. So that the test could
>>>>>>>> be tweaked to inject an error, and ensure the error handling code works as
>>>>>>>> expected
>>>>>>>>
>>>>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post
>>>>>>>>> commit integration tests for SQL, which I suppose includes
>>>>>>>>> DataStoreReadWriteIT
>>>>>>>>>
>>>>>>>>> Let me know if whether or not this is sufficient.
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> Ke
>>>>>>>>>
>>>>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Is it possible to run a Datastore IO integration test to test this
>>>>>>>>> PR?
>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>>>>>
>>>>>>>>> Probably this test can be ran somehow. Though I don't know the
>>>>>>>>> gradle command to run it
>>>>>>>>>
>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>>>>>
>>>>>>>>> Does anyone know how to run this test?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>>>
>>>>>> Software Engineer
>>>>>>
>>>>>> miguel.anzo@wizeline.com
>>>>>>
>>>>>> Remote Office
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> miguel.anzo@wizeline.com
>>>>>
>>>>> Remote Office
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *This email and its contents (including any attachments) are being
>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>> norepresentation is made with respect to any content contained in this
>>>>> email.*
>>>>
>>>>
--
Miguel Angel Anzo Palomo | WIZELINE
Software Engineer
miguel.anzo@wizeline.com
Remote Office
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Command line to run DatstoreIO integration tests for java
Posted by Kyle Weaver <kc...@google.com>.
There's a hard-coded "include" clause in the source. You can change that.
https://github.com/apache/beam/blob/a0fbe00ef12b72ec89672ab32ccc6d6331ca5edd/sdks/java/extensions/sql/build.gradle#L198
On Wed, Aug 18, 2021 at 11:19 AM Andrew Pilloud <ap...@google.com> wrote:
> You might be able to use the '--tests' flag. (This works for unit tests.)
> For example:
>
> ./gradlew :sdks:java:extensions:sql:postCommit
> --tests org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead
>
> On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath <ch...@google.com>
> wrote:
>
>> I don't have the Gradle command to run an individual IT unfortunately. I
>> usually invoke the corresponding Jenkins test suite (PostCommit SQL in this
>> case).
>> Note that there are additional Datastore read/write IT tests that do not
>> involve SQL (and run as a part of Java PostCommit).
>>
>> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
>>
>> Thanks,
>> Cham
>>
>> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <aj...@google.com> wrote:
>>
>>> +Chamikara, +Heejong,
>>>
>>> Do you have any ideas here? I want to verify the library does indeed
>>> produce exceptions as we expect and that we can catch them as done here:
>>> https://github.com/apache/beam/pull/15183
>>>
>>> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
>>> miguel.anzo@wizeline.com> wrote:
>>>
>>>> Hi, I was able to run the integration test locally, but I'm having some
>>>> troubles injecting an error to inject an error here [1] to trigger a
>>>> DatastoreException here [2]. The problem is that if I try to change the
>>>> table or fields in the select query, an exception is raised in [1] when
>>>> parsing the sql to create the BeamRelNode. Would anyone know a way to
>>>> inject an error in the integration test to cause a DatastoreException in
>>>> [2]?
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
>>>> [2]
>>>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>>>>
>>>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
>>>> miguel.anzo@wizeline.com> wrote:
>>>>
>>>>> Hi Alex,
>>>>> My problem trying to locally run that test was that, since I don't
>>>>> have access to the apache-beam-testing GCP project I was having problems
>>>>> getting a datastore instance to read/write from.
>>>>>
>>>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
>>>>>
>>>>>> Miguel, I believe you were having trouble running this test, you
>>>>>> mentioned that you needed a Datastore instance.
>>>>>> Would you mind sharing the errors or any issue you had when you tried
>>>>>> Ke's command?
>>>>>>
>>>>>> Ke, Do we need to specify a datastore instance to run this test?
>>>>>> Where will it read/write from? I assume it will it default to something in
>>>>>> the apache-beam-testing GCP project, as other tests do.
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>
>>>>>>> “Run SQL PostCommit” is essentially running
>>>>>>> “./gradlew :sqlPostCommit” [1]
>>>>>>>
>>>>>>> In your case where you would like to run DataStoreReadWriteIT only,
>>>>>>> it can be simplified [2] to
>>>>>>>
>>>>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>>>>
>>>>>>> Best,
>>>>>>> Ke
>>>>>>>
>>>>>>> [1]
>>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>>>>
>>>>>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>>>>>
>>>>>>> I was hoping for the command line to run it. So that the test could
>>>>>>> be tweaked to inject an error, and ensure the error handling code works as
>>>>>>> expected
>>>>>>>
>>>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post
>>>>>>>> commit integration tests for SQL, which I suppose includes
>>>>>>>> DataStoreReadWriteIT
>>>>>>>>
>>>>>>>> Let me know if whether or not this is sufficient.
>>>>>>>>
>>>>>>>> Best,
>>>>>>>> Ke
>>>>>>>>
>>>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Is it possible to run a Datastore IO integration test to test this
>>>>>>>> PR?
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>>>>
>>>>>>>> Probably this test can be ran somehow. Though I don't know the
>>>>>>>> gradle command to run it
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>>>>
>>>>>>>> Does anyone know how to run this test?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>>
>>>>> Software Engineer
>>>>>
>>>>> miguel.anzo@wizeline.com
>>>>>
>>>>> Remote Office
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>
>>>> Software Engineer
>>>>
>>>> miguel.anzo@wizeline.com
>>>>
>>>> Remote Office
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *This email and its contents (including any attachments) are being sent
>>>> toyou on the condition of confidentiality and may be protected by
>>>> legalprivilege. Access to this email by anyone other than the intended
>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>> immediatelynotify the sender by replying to this message and delete the
>>>> materialimmediately from your system. Any further use, dissemination,
>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>> norepresentation is made with respect to any content contained in this
>>>> email.*
>>>
>>>
Re: Command line to run DatstoreIO integration tests for java
Posted by Andrew Pilloud <ap...@google.com>.
You might be able to use the '--tests' flag. (This works for unit tests.)
For example:
./gradlew :sdks:java:extensions:sql:postCommit
--tests org.apache.beam.sdk.extensions.sql.meta.provider.datastore.DataStoreReadWriteIT.testDataStoreV1SqlWriteRead
On Wed, Aug 18, 2021 at 11:14 AM Chamikara Jayalath <ch...@google.com>
wrote:
> I don't have the Gradle command to run an individual IT unfortunately. I
> usually invoke the corresponding Jenkins test suite (PostCommit SQL in this
> case).
> Note that there are additional Datastore read/write IT tests that do not
> involve SQL (and run as a part of Java PostCommit).
>
> https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
>
> Thanks,
> Cham
>
> On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <aj...@google.com> wrote:
>
>> +Chamikara, +Heejong,
>>
>> Do you have any ideas here? I want to verify the library does indeed
>> produce exceptions as we expect and that we can catch them as done here:
>> https://github.com/apache/beam/pull/15183
>>
>> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
>> miguel.anzo@wizeline.com> wrote:
>>
>>> Hi, I was able to run the integration test locally, but I'm having some
>>> troubles injecting an error to inject an error here [1] to trigger a
>>> DatastoreException here [2]. The problem is that if I try to change the
>>> table or fields in the select query, an exception is raised in [1] when
>>> parsing the sql to create the BeamRelNode. Would anyone know a way to
>>> inject an error in the integration test to cause a DatastoreException in
>>> [2]?
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
>>> [2]
>>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>>>
>>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
>>> miguel.anzo@wizeline.com> wrote:
>>>
>>>> Hi Alex,
>>>> My problem trying to locally run that test was that, since I don't have
>>>> access to the apache-beam-testing GCP project I was having problems getting
>>>> a datastore instance to read/write from.
>>>>
>>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> Miguel, I believe you were having trouble running this test, you
>>>>> mentioned that you needed a Datastore instance.
>>>>> Would you mind sharing the errors or any issue you had when you tried
>>>>> Ke's command?
>>>>>
>>>>> Ke, Do we need to specify a datastore instance to run this test? Where
>>>>> will it read/write from? I assume it will it default to something in
>>>>> the apache-beam-testing GCP project, as other tests do.
>>>>>
>>>>>
>>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>
>>>>>> “Run SQL PostCommit” is essentially running
>>>>>> “./gradlew :sqlPostCommit” [1]
>>>>>>
>>>>>> In your case where you would like to run DataStoreReadWriteIT only,
>>>>>> it can be simplified [2] to
>>>>>>
>>>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>>>
>>>>>> Best,
>>>>>> Ke
>>>>>>
>>>>>> [1]
>>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>>>
>>>>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>>>
>>>>>>
>>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>>>>
>>>>>> I was hoping for the command line to run it. So that the test could
>>>>>> be tweaked to inject an error, and ensure the error handling code works as
>>>>>> expected
>>>>>>
>>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>>
>>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post
>>>>>>> commit integration tests for SQL, which I suppose includes
>>>>>>> DataStoreReadWriteIT
>>>>>>>
>>>>>>> Let me know if whether or not this is sufficient.
>>>>>>>
>>>>>>> Best,
>>>>>>> Ke
>>>>>>>
>>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>>>>>>
>>>>>>> Is it possible to run a Datastore IO integration test to test this
>>>>>>> PR?
>>>>>>>
>>>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>>>
>>>>>>> Probably this test can be ran somehow. Though I don't know the
>>>>>>> gradle command to run it
>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>>>
>>>>>>> Does anyone know how to run this test?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>> --
>>>>
>>>> Miguel Angel Anzo Palomo | WIZELINE
>>>>
>>>> Software Engineer
>>>>
>>>> miguel.anzo@wizeline.com
>>>>
>>>> Remote Office
>>>>
>>>
>>>
>>> --
>>>
>>> Miguel Angel Anzo Palomo | WIZELINE
>>>
>>> Software Engineer
>>>
>>> miguel.anzo@wizeline.com
>>>
>>> Remote Office
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
Re: Command line to run DatstoreIO integration tests for java
Posted by Chamikara Jayalath <ch...@google.com>.
I don't have the Gradle command to run an individual IT unfortunately. I
usually invoke the corresponding Jenkins test suite (PostCommit SQL in this
case).
Note that there are additional Datastore read/write IT tests that do not
involve SQL (and run as a part of Java PostCommit).
https://ci-beam.apache.org/job/beam_PostCommit_Java/7917/testReport/org.apache.beam.sdk.io.gcp.datastore/
Thanks,
Cham
On Tue, Aug 17, 2021 at 5:33 PM Alex Amato <aj...@google.com> wrote:
> +Chamikara, +Heejong,
>
> Do you have any ideas here? I want to verify the library does indeed
> produce exceptions as we expect and that we can catch them as done here:
> https://github.com/apache/beam/pull/15183
>
> On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <
> miguel.anzo@wizeline.com> wrote:
>
>> Hi, I was able to run the integration test locally, but I'm having some
>> troubles injecting an error to inject an error here [1] to trigger a
>> DatastoreException here [2]. The problem is that if I try to change the
>> table or fields in the select query, an exception is raised in [1] when
>> parsing the sql to create the BeamRelNode. Would anyone know a way to
>> inject an error in the integration test to cause a DatastoreException in
>> [2]?
>>
>> [1]
>> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
>> [2]
>> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>>
>> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
>> miguel.anzo@wizeline.com> wrote:
>>
>>> Hi Alex,
>>> My problem trying to locally run that test was that, since I don't have
>>> access to the apache-beam-testing GCP project I was having problems getting
>>> a datastore instance to read/write from.
>>>
>>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
>>>
>>>> Miguel, I believe you were having trouble running this test, you
>>>> mentioned that you needed a Datastore instance.
>>>> Would you mind sharing the errors or any issue you had when you tried
>>>> Ke's command?
>>>>
>>>> Ke, Do we need to specify a datastore instance to run this test? Where
>>>> will it read/write from? I assume it will it default to something in
>>>> the apache-beam-testing GCP project, as other tests do.
>>>>
>>>>
>>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>>
>>>>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit”
>>>>> [1]
>>>>>
>>>>> In your case where you would like to run DataStoreReadWriteIT only, it
>>>>> can be simplified [2] to
>>>>>
>>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>>
>>>>> Best,
>>>>> Ke
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>>
>>>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>>
>>>>>
>>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>>>
>>>>> I was hoping for the command line to run it. So that the test could be
>>>>> tweaked to inject an error, and ensure the error handling code works as
>>>>> expected
>>>>>
>>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>>
>>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post
>>>>>> commit integration tests for SQL, which I suppose includes
>>>>>> DataStoreReadWriteIT
>>>>>>
>>>>>> Let me know if whether or not this is sufficient.
>>>>>>
>>>>>> Best,
>>>>>> Ke
>>>>>>
>>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>>>>>
>>>>>> Is it possible to run a Datastore IO integration test to test this PR?
>>>>>>
>>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>>
>>>>>> Probably this test can be ran somehow. Though I don't know the gradle
>>>>>> command to run it
>>>>>>
>>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>>
>>>>>> Does anyone know how to run this test?
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>
>>> --
>>>
>>> Miguel Angel Anzo Palomo | WIZELINE
>>>
>>> Software Engineer
>>>
>>> miguel.anzo@wizeline.com
>>>
>>> Remote Office
>>>
>>
>>
>> --
>>
>> Miguel Angel Anzo Palomo | WIZELINE
>>
>> Software Engineer
>>
>> miguel.anzo@wizeline.com
>>
>> Remote Office
>>
>>
>>
>>
>>
>>
>>
>>
>> *This email and its contents (including any attachments) are being sent
>> toyou on the condition of confidentiality and may be protected by
>> legalprivilege. Access to this email by anyone other than the intended
>> recipientis unauthorized. If you are not the intended recipient, please
>> immediatelynotify the sender by replying to this message and delete the
>> materialimmediately from your system. Any further use, dissemination,
>> distributionor reproduction of this email is strictly prohibited. Further,
>> norepresentation is made with respect to any content contained in this
>> email.*
>
>
Re: Command line to run DatstoreIO integration tests for java
Posted by Alex Amato <aj...@google.com>.
+Chamikara, +Heejong,
Do you have any ideas here? I want to verify the library does indeed
produce exceptions as we expect and that we can catch them as done here:
https://github.com/apache/beam/pull/15183
On Mon, Aug 16, 2021 at 3:26 PM Miguel Anzo Palomo <mi...@wizeline.com>
wrote:
> Hi, I was able to run the integration test locally, but I'm having some
> troubles injecting an error to inject an error here [1] to trigger a
> DatastoreException here [2]. The problem is that if I try to change the
> table or fields in the select query, an exception is raised in [1] when
> parsing the sql to create the BeamRelNode. Would anyone know a way to
> inject an error in the integration test to cause a DatastoreException in
> [2]?
>
> [1]
> https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
> [2]
> https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
>
> On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <
> miguel.anzo@wizeline.com> wrote:
>
>> Hi Alex,
>> My problem trying to locally run that test was that, since I don't have
>> access to the apache-beam-testing GCP project I was having problems getting
>> a datastore instance to read/write from.
>>
>> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Miguel, I believe you were having trouble running this test, you
>>> mentioned that you needed a Datastore instance.
>>> Would you mind sharing the errors or any issue you had when you tried
>>> Ke's command?
>>>
>>> Ke, Do we need to specify a datastore instance to run this test? Where
>>> will it read/write from? I assume it will it default to something in
>>> the apache-beam-testing GCP project, as other tests do.
>>>
>>>
>>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>>
>>>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit”
>>>> [1]
>>>>
>>>> In your case where you would like to run DataStoreReadWriteIT only, it
>>>> can be simplified [2] to
>>>>
>>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>>
>>>> Best,
>>>> Ke
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>>
>>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>>
>>>>
>>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>>
>>>> I was hoping for the command line to run it. So that the test could be
>>>> tweaked to inject an error, and ensure the error handling code works as
>>>> expected
>>>>
>>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>>
>>>>> Comment the PR with "Run SQL PostCommit” would trigger the post commit
>>>>> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>>>>>
>>>>> Let me know if whether or not this is sufficient.
>>>>>
>>>>> Best,
>>>>> Ke
>>>>>
>>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>>>>
>>>>> Is it possible to run a Datastore IO integration test to test this PR?
>>>>>
>>>>> https://github.com/apache/beam/pull/15183/files
>>>>>
>>>>> Probably this test can be ran somehow. Though I don't know the gradle
>>>>> command to run it
>>>>>
>>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>>
>>>>> Does anyone know how to run this test?
>>>>>
>>>>>
>>>>>
>>>>
>>
>> --
>>
>> Miguel Angel Anzo Palomo | WIZELINE
>>
>> Software Engineer
>>
>> miguel.anzo@wizeline.com
>>
>> Remote Office
>>
>
>
> --
>
> Miguel Angel Anzo Palomo | WIZELINE
>
> Software Engineer
>
> miguel.anzo@wizeline.com
>
> Remote Office
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
Re: Command line to run DatstoreIO integration tests for java
Posted by Miguel Anzo Palomo <mi...@wizeline.com>.
Hi, I was able to run the integration test locally, but I'm having some
troubles injecting an error to inject an error here [1] to trigger a
DatastoreException here [2]. The problem is that if I try to change the
table or fields in the select query, an exception is raised in [1] when
parsing the sql to create the BeamRelNode. Would anyone know a way to
inject an error in the integration test to cause a DatastoreException in
[2]?
[1]
https://github.com/apache/beam/blob/ce406c69d2d06e5c5b659fa259253d5cc64249e7/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java#L105
[2]
https://github.com/apache/beam/blob/089ca7a62be86ad7865308f2cbd676bb18d7d648/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L927
On Tue, Aug 10, 2021 at 3:25 PM Miguel Anzo Palomo <mi...@wizeline.com>
wrote:
> Hi Alex,
> My problem trying to locally run that test was that, since I don't have
> access to the apache-beam-testing GCP project I was having problems getting
> a datastore instance to read/write from.
>
> On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
>
>> Miguel, I believe you were having trouble running this test, you
>> mentioned that you needed a Datastore instance.
>> Would you mind sharing the errors or any issue you had when you tried
>> Ke's command?
>>
>> Ke, Do we need to specify a datastore instance to run this test? Where
>> will it read/write from? I assume it will it default to something in
>> the apache-beam-testing GCP project, as other tests do.
>>
>>
>> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>>
>>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit”
>>> [1]
>>>
>>> In your case where you would like to run DataStoreReadWriteIT only, it
>>> can be simplified [2] to
>>>
>>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>>
>>> Best,
>>> Ke
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>>
>>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>>
>>>
>>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>>
>>> I was hoping for the command line to run it. So that the test could be
>>> tweaked to inject an error, and ensure the error handling code works as
>>> expected
>>>
>>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>>
>>>> Comment the PR with "Run SQL PostCommit” would trigger the post commit
>>>> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>>>>
>>>> Let me know if whether or not this is sufficient.
>>>>
>>>> Best,
>>>> Ke
>>>>
>>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>>>
>>>> Is it possible to run a Datastore IO integration test to test this PR?
>>>>
>>>> https://github.com/apache/beam/pull/15183/files
>>>>
>>>> Probably this test can be ran somehow. Though I don't know the gradle
>>>> command to run it
>>>>
>>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>>
>>>> Does anyone know how to run this test?
>>>>
>>>>
>>>>
>>>
>
> --
>
> Miguel Angel Anzo Palomo | WIZELINE
>
> Software Engineer
>
> miguel.anzo@wizeline.com
>
> Remote Office
>
--
Miguel Angel Anzo Palomo | WIZELINE
Software Engineer
miguel.anzo@wizeline.com
Remote Office
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Command line to run DatstoreIO integration tests for java
Posted by Miguel Anzo Palomo <mi...@wizeline.com>.
Hi Alex,
My problem trying to locally run that test was that, since I don't have
access to the apache-beam-testing GCP project I was having problems getting
a datastore instance to read/write from.
On Tue, Aug 10, 2021 at 1:08 PM Alex Amato <aj...@google.com> wrote:
> Miguel, I believe you were having trouble running this test, you mentioned
> that you needed a Datastore instance.
> Would you mind sharing the errors or any issue you had when you tried Ke's
> command?
>
> Ke, Do we need to specify a datastore instance to run this test? Where
> will it read/write from? I assume it will it default to something in
> the apache-beam-testing GCP project, as other tests do.
>
>
> On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
>
>> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1]
>>
>> In your case where you would like to run DataStoreReadWriteIT only, it
>> can be simplified [2] to
>>
>> "./gradlew :sdks:java:extensions:sql:postCommit”
>>
>> Best,
>> Ke
>>
>> [1]
>> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>>
>> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>>
>>
>> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>>
>> I was hoping for the command line to run it. So that the test could be
>> tweaked to inject an error, and ensure the error handling code works as
>> expected
>>
>> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>>
>>> Comment the PR with "Run SQL PostCommit” would trigger the post commit
>>> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>>>
>>> Let me know if whether or not this is sufficient.
>>>
>>> Best,
>>> Ke
>>>
>>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>>
>>> Is it possible to run a Datastore IO integration test to test this PR?
>>>
>>> https://github.com/apache/beam/pull/15183/files
>>>
>>> Probably this test can be ran somehow. Though I don't know the gradle
>>> command to run it
>>>
>>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>>
>>> Does anyone know how to run this test?
>>>
>>>
>>>
>>
--
Miguel Angel Anzo Palomo | WIZELINE
Software Engineer
miguel.anzo@wizeline.com
Remote Office
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: Command line to run DatstoreIO integration tests for java
Posted by Alex Amato <aj...@google.com>.
Miguel, I believe you were having trouble running this test, you mentioned
that you needed a Datastore instance.
Would you mind sharing the errors or any issue you had when you tried Ke's
command?
Ke, Do we need to specify a datastore instance to run this test? Where will
it read/write from? I assume it will it default to something in
the apache-beam-testing GCP project, as other tests do.
On Thu, Jul 29, 2021 at 6:16 PM Ke Wu <ke...@gmail.com> wrote:
> “Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1]
>
> In your case where you would like to run DataStoreReadWriteIT only, it can
> be simplified [2] to
>
> "./gradlew :sdks:java:extensions:sql:postCommit”
>
> Best,
> Ke
>
> [1]
> https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41
>
> [2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190
>
>
> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>
> I was hoping for the command line to run it. So that the test could be
> tweaked to inject an error, and ensure the error handling code works as
> expected
>
> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
>
>> Comment the PR with "Run SQL PostCommit” would trigger the post commit
>> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>>
>> Let me know if whether or not this is sufficient.
>>
>> Best,
>> Ke
>>
>> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>>
>> Is it possible to run a Datastore IO integration test to test this PR?
>>
>> https://github.com/apache/beam/pull/15183/files
>>
>> Probably this test can be ran somehow. Though I don't know the gradle
>> command to run it
>>
>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>>
>> Does anyone know how to run this test?
>>
>>
>>
>
Re: Command line to run DatstoreIO integration tests for java
Posted by Ke Wu <ke...@gmail.com>.
“Run SQL PostCommit” is essentially running “./gradlew :sqlPostCommit” [1]
In your case where you would like to run DataStoreReadWriteIT only, it can be simplified [2] to
"./gradlew :sdks:java:extensions:sql:postCommit”
Best,
Ke
[1] https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41 <https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_SQL.groovy#L41>
[2] https://github.com/apache/beam/blob/master/build.gradle.kts#L190 <https://github.com/apache/beam/blob/master/build.gradle.kts#L190>
> On Jul 29, 2021, at 9:58 AM, Alex Amato <aj...@google.com> wrote:
>
> I was hoping for the command line to run it. So that the test could be tweaked to inject an error, and ensure the error handling code works as expected
>
> On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke.wu.cs@gmail.com <ma...@gmail.com>> wrote:
> Comment the PR with "Run SQL PostCommit” would trigger the post commit integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>
> Let me know if whether or not this is sufficient.
>
> Best,
> Ke
>
>> On Jul 28, 2021, at 12:20 PM, Alex Amato <ajamato@google.com <ma...@google.com>> wrote:
>>
>> Is it possible to run a Datastore IO integration test to test this PR?
>>
>> https://github.com/apache/beam/pull/15183/files <https://github.com/apache/beam/pull/15183/files>
>>
>> Probably this test can be ran somehow. Though I don't know the gradle command to run it
>> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java <https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java>
>>
>> Does anyone know how to run this test?
>
Re: Command line to run DatstoreIO integration tests for java
Posted by Alex Amato <aj...@google.com>.
I was hoping for the command line to run it. So that the test could be
tweaked to inject an error, and ensure the error handling code works as
expected
On Wed, Jul 28, 2021 at 8:34 PM Ke Wu <ke...@gmail.com> wrote:
> Comment the PR with "Run SQL PostCommit” would trigger the post commit
> integration tests for SQL, which I suppose includes DataStoreReadWriteIT
>
> Let me know if whether or not this is sufficient.
>
> Best,
> Ke
>
> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>
> Is it possible to run a Datastore IO integration test to test this PR?
>
> https://github.com/apache/beam/pull/15183/files
>
> Probably this test can be ran somehow. Though I don't know the gradle
> command to run it
>
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java
>
> Does anyone know how to run this test?
>
>
>
Re: Command line to run DatstoreIO integration tests for java
Posted by Ke Wu <ke...@gmail.com>.
Comment the PR with "Run SQL PostCommit” would trigger the post commit integration tests for SQL, which I suppose includes DataStoreReadWriteIT
Let me know if whether or not this is sufficient.
Best,
Ke
> On Jul 28, 2021, at 12:20 PM, Alex Amato <aj...@google.com> wrote:
>
> Is it possible to run a Datastore IO integration test to test this PR?
>
> https://github.com/apache/beam/pull/15183/files <https://github.com/apache/beam/pull/15183/files>
>
> Probably this test can be ran somehow. Though I don't know the gradle command to run it
> https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java <https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datastore/DataStoreReadWriteIT.java>
>
> Does anyone know how to run this test?