You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ritwik Dutta via dev <de...@beam.apache.org> on 2024/05/22 20:21:27 UTC
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
any response yet? No one has answers? I left a stackoverflow bounty on the question
Using external methods is pretty important
On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <rd...@yahoo.com> wrote:
Hi, I wrote the following question here.It would be really helpful also, if you can also update your documentation on Using Test Fakes in different Situations. It was very light documentation. Please provide more explanation and examples. https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
Question: Java Apache Beam, mock external Clients initialized in @Setup method of DoFn with Constructors variableshttps://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
Thanks,
-Ritwik Dutta 734-262-4285
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
Posted by Steven van Rossum via dev <de...@beam.apache.org>.
A recurring pattern, although not uniformly named, for I/O connectors is to
provide a method on the PTransform named something along the lines of
withClient, withClientFactory, withClientBuilderFactory,
withClientProvider, etc. which stores the configuration the client is to be
instantiated with and a method to perform that instantiation.
DoFns instantiated by that PTransform should store the instance of that
client configuration class and call the method to instantiate the client
(e.g. buildClient).
That doesn't make the process of defining mocks or fakes any easier since
that complexity is inherent to the mock framework or the complexity of the
service being faked, but it provides a method to customize the client
instantiation outside of the PTransform or DoFn.
Have a look at
https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub,
PubsubIOTest.java and PubsubTestClient.java might be relevant examples.
On Wed, May 29, 2024 at 1:41 AM Robert Bradshaw via dev <de...@beam.apache.org>
wrote:
> You could let your @Setup method initialize your serviceAPI if (and
> only if) it's null. For your tests, pre-initialize it with a Fake, and
> otherwise the real thing will get used.
>
> On Sat, May 25, 2024 at 1:27 PM Ritwik Dutta via dev
> <de...@beam.apache.org> wrote:
> >
> > Hi Hu,
> >
> > Thank you,
> > I am testing the serviceAPI call in separate unit tests already. Let me
> rephrase (serviceAPI should be renamed to serviceLayer, it doesn't
> necessarily have to be a restful API with an http endpoint)
> > The question is how, do we create Fakes for items with difficult
> constructors.
> >
> > At least provide examples, or more explanation in Apache dcoumentation.
> >
> > Feel free to reply directly in stackoverflow, so other community members
> can reply
> >
> > Thanks!
> >
> > -Ritwik Dutta
> > 734-262-4285 <(734)%20262-4285>
> >
> > On Saturday, May 25, 2024 at 07:12:23 AM PDT, XQ Hu <xq...@google.com>
> wrote:
> >
> >
> > I am not sure which part you want to test. If the processData part
> should be tested, you could refactor the code without use any Beam specific
> code and test the processing data logic.
> >
> > From your example, it seems that you are calling some APIs, we recently
> added a new Web API IO:
> https://beam.apache.org/documentation/io/built-in/webapis/, which
> provides a way to test.
> >
> > On Wed, May 22, 2024 at 5:06 PM Ritwik Dutta via dev <
> dev@beam.apache.org> wrote:
> >
> > any response yet? No one has answers? I left a stackoverflow bounty on
> the question
> >
> > Using external methods is pretty important
> >
> > On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <
> rdutta23@yahoo.com> wrote:
> >
> >
> > Hi,
> > I wrote the following question here.
> > It would be really helpful also, if you can also update your
> documentation on Using Test Fakes in different Situations. It was very
> light documentation. Please provide more explanation and examples.
> https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
> >
> >
> > Question: Java Apache Beam, mock external Clients initialized in @Setup
> method of DoFn with Constructors variables
> >
> https://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
> >
> > Thanks,
> >
> > -Ritwik Dutta
> > 734-262-4285 <(734)%20262-4285>
>
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
You could let your @Setup method initialize your serviceAPI if (and
only if) it's null. For your tests, pre-initialize it with a Fake, and
otherwise the real thing will get used.
On Sat, May 25, 2024 at 1:27 PM Ritwik Dutta via dev
<de...@beam.apache.org> wrote:
>
> Hi Hu,
>
> Thank you,
> I am testing the serviceAPI call in separate unit tests already. Let me rephrase (serviceAPI should be renamed to serviceLayer, it doesn't necessarily have to be a restful API with an http endpoint)
> The question is how, do we create Fakes for items with difficult constructors.
>
> At least provide examples, or more explanation in Apache dcoumentation.
>
> Feel free to reply directly in stackoverflow, so other community members can reply
>
> Thanks!
>
> -Ritwik Dutta
> 734-262-4285
>
> On Saturday, May 25, 2024 at 07:12:23 AM PDT, XQ Hu <xq...@google.com> wrote:
>
>
> I am not sure which part you want to test. If the processData part should be tested, you could refactor the code without use any Beam specific code and test the processing data logic.
>
> From your example, it seems that you are calling some APIs, we recently added a new Web API IO: https://beam.apache.org/documentation/io/built-in/webapis/, which provides a way to test.
>
> On Wed, May 22, 2024 at 5:06 PM Ritwik Dutta via dev <de...@beam.apache.org> wrote:
>
> any response yet? No one has answers? I left a stackoverflow bounty on the question
>
> Using external methods is pretty important
>
> On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <rd...@yahoo.com> wrote:
>
>
> Hi,
> I wrote the following question here.
> It would be really helpful also, if you can also update your documentation on Using Test Fakes in different Situations. It was very light documentation. Please provide more explanation and examples. https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
>
>
> Question: Java Apache Beam, mock external Clients initialized in @Setup method of DoFn with Constructors variables
> https://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
>
> Thanks,
>
> -Ritwik Dutta
> 734-262-4285
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
Posted by Ritwik Dutta via dev <de...@beam.apache.org>.
Hi Hu,
Thank you,I am testing the serviceAPI call in separate unit tests already. Let me rephrase (serviceAPI should be renamed to serviceLayer, it doesn't necessarily have to be a restful API with an http endpoint)The question is how, do we create Fakes for items with difficult constructors.
At least provide examples, or more explanation in Apache dcoumentation.
Feel free to reply directly in stackoverflow, so other community members can reply
Thanks!
-Ritwik Dutta 734-262-4285
On Saturday, May 25, 2024 at 07:12:23 AM PDT, XQ Hu <xq...@google.com> wrote:
I am not sure which part you want to test. If the processData part should be tested, you could refactor the code without use any Beam specific code and test the processing data logic.
From your example, it seems that you are calling some APIs, we recently added a new Web API IO: https://beam.apache.org/documentation/io/built-in/webapis/, which provides a way to test.
On Wed, May 22, 2024 at 5:06 PM Ritwik Dutta via dev <de...@beam.apache.org> wrote:
any response yet? No one has answers? I left a stackoverflow bounty on the question
Using external methods is pretty important
On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <rd...@yahoo.com> wrote:
Hi, I wrote the following question here.It would be really helpful also, if you can also update your documentation on Using Test Fakes in different Situations. It was very light documentation. Please provide more explanation and examples. https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
Question: Java Apache Beam, mock external Clients initialized in @Setup method of DoFn with Constructors variableshttps://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
Thanks,
-Ritwik Dutta 734-262-4285
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
Posted by XQ Hu via dev <de...@beam.apache.org>.
I am not sure which part you want to test. If the processData part should
be tested, you could refactor the code without use any Beam specific code
and test the processing data logic.
From your example, it seems that you are calling some APIs, we recently
added a new Web API IO:
https://beam.apache.org/documentation/io/built-in/webapis/,
which provides a way to test.
On Wed, May 22, 2024 at 5:06 PM Ritwik Dutta via dev <de...@beam.apache.org>
wrote:
> any response yet? No one has answers? I left a stackoverflow bounty on the
> question
>
> Using external methods is pretty important
>
> On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <
> rdutta23@yahoo.com> wrote:
>
>
> Hi,
> I wrote the following question here.
> It would be really helpful also, if you can also update your documentation
> on Using Test Fakes in different Situations. It was very light
> documentation. Please provide more explanation and examples.
> https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
>
>
> *Question: *Java Apache Beam, mock external Clients initialized in @Setup
> method of DoFn with Constructors variables
>
> https://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
>
> Thanks,
>
> -Ritwik Dutta
> 734-262-4285 <(734)%20262-4285>
>
Re: Question: Java Apache Beam, mock external Clients initialized in Setup
Posted by XQ Hu via user <us...@beam.apache.org>.
I am not sure which part you want to test. If the processData part should
be tested, you could refactor the code without use any Beam specific code
and test the processing data logic.
From your example, it seems that you are calling some APIs, we recently
added a new Web API IO:
https://beam.apache.org/documentation/io/built-in/webapis/,
which provides a way to test.
On Wed, May 22, 2024 at 5:06 PM Ritwik Dutta via dev <de...@beam.apache.org>
wrote:
> any response yet? No one has answers? I left a stackoverflow bounty on the
> question
>
> Using external methods is pretty important
>
> On Sunday, May 12, 2024 at 11:52:25 AM PDT, Ritwik Dutta <
> rdutta23@yahoo.com> wrote:
>
>
> Hi,
> I wrote the following question here.
> It would be really helpful also, if you can also update your documentation
> on Using Test Fakes in different Situations. It was very light
> documentation. Please provide more explanation and examples.
> https://beam.apache.org/documentation/io/testing/#:~:text=non%2DBeam%20client.-,Use%20fakes,-Instead%20of%20using
>
>
> *Question: *Java Apache Beam, mock external Clients initialized in @Setup
> method of DoFn with Constructors variables
>
> https://stackoverflow.com/questions/78468953/java-apache-beam-mock-external-clients-initialized-in-setup-method-of-dofn-wit
>
> Thanks,
>
> -Ritwik Dutta
> 734-262-4285 <(734)%20262-4285>
>