You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2019/01/17 18:48:54 UTC
Where should I locate shared/common Java Code (Java SDK and Java RH
both use)?
I would like to reuse the State Sampling classes from the Dataflow Runner
Harness in the Beam Java SDK. I created a Refactoring plan
<https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
removed the Dataflow references from the classes in PR/7507
<https://github.com/apache/beam/pull/7507>.
SInce the Java SDK cannot depend on the Dataflow Runner Harness, then I can
do one of these options
- Create a new gradle project/folder and locate the code in a shared
location?
- i.e. create a new: beam/*common*/java/*extensions/state-sampling*
- Move the code in a Java SDK projects, which will work since the runner
harness can depend on the SDK?
- i.e.reuse the existing: beam/*sdks*/java/*extensions/state-sampling*
Re: Where should I locate shared/common Java Code (Java SDK and Java
RH both use)?
Posted by Alex Amato <aj...@google.com>.
Cool, thanks! Sounds like a good place to put it. Appreciate the pointer :)
On Thu, Jan 17, 2019 at 3:47 PM Kenneth Knowles <kl...@google.com> wrote:
> I think runners/core-java fits in this case. It started life as a place
> for shared functionality to all JVM-based runners, but much of that also
> turns out to apply to the Java SDK harness. I see it is already used in the
> harness.
>
> Kenn
>
> On Thu, Jan 17, 2019 at 1:55 PM Alex Amato <aj...@google.com> wrote:
>
>> any Java SDK Harness coult use*
>>
>> On Thu, Jan 17, 2019 at 1:22 PM Alex Amato <aj...@google.com> wrote:
>>
>>> Just the SDK harness, it doesn't need to be part of the user APIs or
>>> anything. But I want to offer it as a library that any Java SDK could use
>>> to enable state sampling as a low weight estimation of execution times,
>>> which is useful to implement the execution time metrics. So I don't see a
>>> need to put it on the core SDK.
>>>
>>> Then I think that our options would be:
>>>
>>> - create a new: beam/java/*extensions/state-sampling*
>>> - reuse the existing: beam/*sdks*/java/*extensions/state-sampling*
>>>
>>>
>>> On Thu, Jan 17, 2019 at 10:59 AM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Do you want to use it in the core SDK or the SDK harness? Does the
>>>> state sampling code depend on the core SDK?
>>>>
>>>> I'd really like to avoid layers with names like "common". Just delete
>>>> that layer of the directory and the path hierarchy has exactly the same
>>>> meaning, but is more concise and clear.
>>>>
>>>> Kenn
>>>>
>>>> On Thu, Jan 17, 2019 at 10:49 AM Alex Amato <aj...@google.com> wrote:
>>>>
>>>>> I would like to reuse the State Sampling classes from the Dataflow
>>>>> Runner Harness in the Beam Java SDK. I created a Refactoring plan
>>>>> <https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
>>>>> removed the Dataflow references from the classes in PR/7507
>>>>> <https://github.com/apache/beam/pull/7507>.
>>>>>
>>>>> SInce the Java SDK cannot depend on the Dataflow Runner Harness, then
>>>>> I can do one of these options
>>>>>
>>>>> - Create a new gradle project/folder and locate the code in a
>>>>> shared location?
>>>>> - i.e. create a new: beam/*common*/java/
>>>>> *extensions/state-sampling*
>>>>> - Move the code in a Java SDK projects, which will work since the
>>>>> runner harness can depend on the SDK?
>>>>> - i.e.reuse the existing: beam/*sdks*/java/
>>>>> *extensions/state-sampling*
>>>>>
>>>>>
Re: Where should I locate shared/common Java Code (Java SDK and Java
RH both use)?
Posted by Kenneth Knowles <kl...@google.com>.
I think runners/core-java fits in this case. It started life as a place for
shared functionality to all JVM-based runners, but much of that also turns
out to apply to the Java SDK harness. I see it is already used in the
harness.
Kenn
On Thu, Jan 17, 2019 at 1:55 PM Alex Amato <aj...@google.com> wrote:
> any Java SDK Harness coult use*
>
> On Thu, Jan 17, 2019 at 1:22 PM Alex Amato <aj...@google.com> wrote:
>
>> Just the SDK harness, it doesn't need to be part of the user APIs or
>> anything. But I want to offer it as a library that any Java SDK could use
>> to enable state sampling as a low weight estimation of execution times,
>> which is useful to implement the execution time metrics. So I don't see a
>> need to put it on the core SDK.
>>
>> Then I think that our options would be:
>>
>> - create a new: beam/java/*extensions/state-sampling*
>> - reuse the existing: beam/*sdks*/java/*extensions/state-sampling*
>>
>>
>> On Thu, Jan 17, 2019 at 10:59 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Do you want to use it in the core SDK or the SDK harness? Does the state
>>> sampling code depend on the core SDK?
>>>
>>> I'd really like to avoid layers with names like "common". Just delete
>>> that layer of the directory and the path hierarchy has exactly the same
>>> meaning, but is more concise and clear.
>>>
>>> Kenn
>>>
>>> On Thu, Jan 17, 2019 at 10:49 AM Alex Amato <aj...@google.com> wrote:
>>>
>>>> I would like to reuse the State Sampling classes from the Dataflow
>>>> Runner Harness in the Beam Java SDK. I created a Refactoring plan
>>>> <https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
>>>> removed the Dataflow references from the classes in PR/7507
>>>> <https://github.com/apache/beam/pull/7507>.
>>>>
>>>> SInce the Java SDK cannot depend on the Dataflow Runner Harness, then I
>>>> can do one of these options
>>>>
>>>> - Create a new gradle project/folder and locate the code in a
>>>> shared location?
>>>> - i.e. create a new: beam/*common*/java/
>>>> *extensions/state-sampling*
>>>> - Move the code in a Java SDK projects, which will work since the
>>>> runner harness can depend on the SDK?
>>>> - i.e.reuse the existing: beam/*sdks*/java/
>>>> *extensions/state-sampling*
>>>>
>>>>
Re: Where should I locate shared/common Java Code (Java SDK and Java
RH both use)?
Posted by Alex Amato <aj...@google.com>.
any Java SDK Harness coult use*
On Thu, Jan 17, 2019 at 1:22 PM Alex Amato <aj...@google.com> wrote:
> Just the SDK harness, it doesn't need to be part of the user APIs or
> anything. But I want to offer it as a library that any Java SDK could use
> to enable state sampling as a low weight estimation of execution times,
> which is useful to implement the execution time metrics. So I don't see a
> need to put it on the core SDK.
>
> Then I think that our options would be:
>
> - create a new: beam/java/*extensions/state-sampling*
> - reuse the existing: beam/*sdks*/java/*extensions/state-sampling*
>
>
> On Thu, Jan 17, 2019 at 10:59 AM Kenneth Knowles <kl...@google.com> wrote:
>
>> Do you want to use it in the core SDK or the SDK harness? Does the state
>> sampling code depend on the core SDK?
>>
>> I'd really like to avoid layers with names like "common". Just delete
>> that layer of the directory and the path hierarchy has exactly the same
>> meaning, but is more concise and clear.
>>
>> Kenn
>>
>> On Thu, Jan 17, 2019 at 10:49 AM Alex Amato <aj...@google.com> wrote:
>>
>>> I would like to reuse the State Sampling classes from the Dataflow
>>> Runner Harness in the Beam Java SDK. I created a Refactoring plan
>>> <https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
>>> removed the Dataflow references from the classes in PR/7507
>>> <https://github.com/apache/beam/pull/7507>.
>>>
>>> SInce the Java SDK cannot depend on the Dataflow Runner Harness, then I
>>> can do one of these options
>>>
>>> - Create a new gradle project/folder and locate the code in a shared
>>> location?
>>> - i.e. create a new: beam/*common*/java/
>>> *extensions/state-sampling*
>>> - Move the code in a Java SDK projects, which will work since the
>>> runner harness can depend on the SDK?
>>> - i.e.reuse the existing: beam/*sdks*/java/
>>> *extensions/state-sampling*
>>>
>>>
Re: Where should I locate shared/common Java Code (Java SDK and Java
RH both use)?
Posted by Alex Amato <aj...@google.com>.
Just the SDK harness, it doesn't need to be part of the user APIs or
anything. But I want to offer it as a library that any Java SDK could use
to enable state sampling as a low weight estimation of execution times,
which is useful to implement the execution time metrics. So I don't see a
need to put it on the core SDK.
Then I think that our options would be:
- create a new: beam/java/*extensions/state-sampling*
- reuse the existing: beam/*sdks*/java/*extensions/state-sampling*
On Thu, Jan 17, 2019 at 10:59 AM Kenneth Knowles <kl...@google.com> wrote:
> Do you want to use it in the core SDK or the SDK harness? Does the state
> sampling code depend on the core SDK?
>
> I'd really like to avoid layers with names like "common". Just delete that
> layer of the directory and the path hierarchy has exactly the same meaning,
> but is more concise and clear.
>
> Kenn
>
> On Thu, Jan 17, 2019 at 10:49 AM Alex Amato <aj...@google.com> wrote:
>
>> I would like to reuse the State Sampling classes from the Dataflow Runner
>> Harness in the Beam Java SDK. I created a Refactoring plan
>> <https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
>> removed the Dataflow references from the classes in PR/7507
>> <https://github.com/apache/beam/pull/7507>.
>>
>> SInce the Java SDK cannot depend on the Dataflow Runner Harness, then I
>> can do one of these options
>>
>> - Create a new gradle project/folder and locate the code in a shared
>> location?
>> - i.e. create a new: beam/*common*/java/*extensions/state-sampling*
>> - Move the code in a Java SDK projects, which will work since the
>> runner harness can depend on the SDK?
>> - i.e.reuse the existing: beam/*sdks*/java/
>> *extensions/state-sampling*
>>
>>
Re: Where should I locate shared/common Java Code (Java SDK and Java
RH both use)?
Posted by Kenneth Knowles <kl...@google.com>.
Do you want to use it in the core SDK or the SDK harness? Does the state
sampling code depend on the core SDK?
I'd really like to avoid layers with names like "common". Just delete that
layer of the directory and the path hierarchy has exactly the same meaning,
but is more concise and clear.
Kenn
On Thu, Jan 17, 2019 at 10:49 AM Alex Amato <aj...@google.com> wrote:
> I would like to reuse the State Sampling classes from the Dataflow Runner
> Harness in the Beam Java SDK. I created a Refactoring plan
> <https://docs.google.com/document/d/1OlAJf4T_CTL9WRH8lP8uQOfLjWYfm8IpRXSe38g34k4/edit#>and
> removed the Dataflow references from the classes in PR/7507
> <https://github.com/apache/beam/pull/7507>.
>
> SInce the Java SDK cannot depend on the Dataflow Runner Harness, then I
> can do one of these options
>
> - Create a new gradle project/folder and locate the code in a shared
> location?
> - i.e. create a new: beam/*common*/java/*extensions/state-sampling*
> - Move the code in a Java SDK projects, which will work since the
> runner harness can depend on the SDK?
> - i.e.reuse the existing: beam/*sdks*/java/
> *extensions/state-sampling*
>
>