You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pirk.apache.org by Ellison Anne Williams <ea...@gmail.com> on 2016/07/17 15:57:26 UTC
PIRK-7 -- JMH license issues
Suneel -- Thanks for creating the JIRA issue and pointing out the licensing
problems. I see that JMH is under the GNU GPL2 (
http://openjdk.java.net/legal/) which is not compatible with the Apache
license (http://www.apache.org/legal/resolved.html).
It appears that Flink just removed the benchmarking code instead of
re-porting it to another option.
I would like us to port it to another license-compatible benchmarking
framework such as Google Caliper (or something similar) instead of removing
the code as the benchmarking is important for encryption optimization.
Thoughts?
Thanks!
Ellison Anne
Re: PIRK-7 -- JMH license issues
Posted by Ellison Anne Williams <ea...@gmail.com>.
+1
So, folks will have to come to the repo and pull the code directly if they
would like to run the benchmarks as it will not be packed in the artifacts.
Let's just go ahead and update the pom accordingly (we already have
pom-benchmarks.xml that will build for benchmarking).
On Sat, Jul 23, 2016 at 1:47 AM, Suneel Marthi <su...@gmail.com>
wrote:
> +1
>
> On Sat, Jul 23, 2016 at 12:01 AM, Jacob Wilder <
> jacobwilder.opensource@gmail.com> wrote:
>
> > I'll put forward my vote that we stick with JMH and separate out any
> > benchmarks that depend on it if/when we distribute binary artifacts. JMH
> > seems to be better maintained (starting from the fact that it's possible
> to
> > use it without needing to modify what you get from the source repo) and
> it
> > seems to be easier to use.
> >
> > If nobody has any qualms with it I will close my pull request and we'll
> > keep using JMH.
> >
> > —
> > Jacob WIlder
> >
> > On Thu, Jul 21, 2016 at 7:23 AM, Tim Ellison <t....@gmail.com>
> > wrote:
> >
> > > On 21/07/16 06:47, Suneel Marthi wrote:
> > > > JMH license is not compatible with Apache and hence this thread.
> > > > But if JMH is only being used for tests and if we don't package the
> > tests
> > > > into project artifacts on Apache mirrors, I am thinking we shuld be
> > fine
> > > to
> > > > leave JMH as is.
> > >
> > > It gets a bit tricky, so here goes:
> > > - the Pirk tests are Apache licensed, so obviously they are ok to
> > > package and redistribute.
> > >
> > > - the JMH library itself is licensed as GPLv2+classpath exception.
> > > This cannot be redistributed by Pirk as all Apache software must be
> > > distributed under the Apache License 2.0.
> > >
> > > - it is ok for Pirk to have a dependency on tools (such as gpg2 for
> > > artefact signing), platforms (such as Java), and /optional/ runtime
> > > plug-ins that have an 'incompatible' license. These are provided by
> the
> > > end-user, not the Pirk project.
> > >
> > > - where Pirk chooses to bundle the output from tools, you have to
> check
> > > the terms that come with each tool to know if there are any
> restrictions.
> > >
> > > > I defer to Tim or Joe to make a call here.
> > >
> > > I see no problem with continuing to use JMH for benchmarking tests.
> > >
> > > Like all Apache projects, our primary distribution will be source code.
> > > When it comes to creating a convenience binary artefact we will audit
> > > it to check there it does not include incompatibly licensed binaries.
> > >
> > > HTH
> > >
> > > Regards,
> > > Tim
> > >
> > > > On Wed, Jul 20, 2016 at 6:00 PM, Jacob Wilder <
> > > > jacobwilder.opensource@gmail.com> wrote:
> > > >
> > > >> Returning to the JMH discussion, I spent a short period of time
> > > integrating
> > > >> Google Caliper and much more time figuring out how to run it.
> > > >>
> > > >> The answer is to use: `java -cp
> > > >> caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
> > > >> com.google.caliper.runner.CaliperMain
> > > >> org.apache.pirk.benchmark.PaillierBenchmark`
> > > >>
> > > >> You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get
> > > Caliper
> > > >> to compile I had to change their pom file to bump both
> > com.google.dagger
> > > >> dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have
> only a
> > > >> vague idea why this makes a difference.
> > > >>
> > > >> At the same time, I've looked around and it looks like we are fine
> to
> > > >> continue using JMH (which, if I am to extrapolate off of my
> experience
> > > in
> > > >> the past two days) is easier to use. There are other Apache projects
> > > that
> > > >> continue to use JMH under their classpath exemption (including
> > > dropwizard's
> > > >> Metrics). Can we articulate a reason why we must switch and why it
> > > would be
> > > >> better to? I'm totally fine with cancelling this PR.
> > > >>
> > > >> —
> > > >> Jacob WIlder
> > > >>
> > > >> On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <smarthi@apache.org
> >
> > > >> wrote:
> > > >>
> > > >>> ....and we shuld move the Beam and Streaming integration
> discussions
> > > to
> > > >> a
> > > >>> different thread.
> > > >>>
> > > >>> On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <
> smarthi@apache.org>
> > > >>> wrote:
> > > >>>
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
> > > >>> chris.harris010@gmail.com>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> I agree separating the benchmarking code out might be a good
> idea.
> > > >>> Maybe
> > > >>>>> we want to make this a multi-module project and the benchmarking
> > be a
> > > >>>>> submodule? Won't JMH still need to included via the child pom in
> > the
> > > >>>>> benchmarking though; I don't think it can be marked as "provided"
> > or
> > > >>>>> anything, right? Glad to hear if you had a different thought on
> how
> > > to
> > > >>> go
> > > >>>>> with this though.
> > > >>>>>
> > > >>>>> Maybe it's good to create submodules for all the adapters (eg.
> > Spark,
> > > >>> MR,
> > > >>>>> Flink, Storm etc...) too? Then we won't have just one jar
> needing
> > to
> > > >>>>> carry
> > > >>>>> around all those dependencies that aren't used (and reduce
> > potential
> > > >>>>> version conflicts). I don't know if this would shake things up
> too
> > > >> much
> > > >>>>> to
> > > >>>>> do this restructuring now, or if it is better to do now before
> we
> > > >> move
> > > >>>>> too
> > > >>>>> far down the line. We'd have to look at what's Pirk "core" vs
> not,
> > > >> but
> > > >>> I
> > > >>>>> don't think that's too tough to discern right now. Most of the
> > > >> adapter
> > > >>>>> code is in org.apache.pirk.responder.
> > > >>>>>
> > > >>>>
> > > >>>> I have a suggestion here. Its not a good idea to be having
> > submodules
> > > >> for
> > > >>>> every streaming engine that's out there. In the long run, its
> gonna
> > be
> > > >> a
> > > >>>> maintenance and compatibility nightmare with every release of
> Flink,
> > > >>> Spark
> > > >>>> etc...
> > > >>>>
> > > >>>> I am guilty of having wasted the last 2 yrs of my life (and fellow
> > > >>>> committers time) in adding support for Spark, Flink, H2O on the
> > Apache
> > > >>>> Mahout project as distributed backend engines.
> > > >>>>
> > > >>>> I wish now that Apache Beam was around in 2014, I would have to
> then
> > > >> just
> > > >>>> support Beam and be abstracted away from all of the other
> Streaming
> > > >>>> frameworks.
> > > >>>>
> > > >>>> As a new project, I would suggest that we look into integrating
> with
> > > >> Beam
> > > >>>> and keep the codebase lean and not bloat the project with all the
> > > >>> different
> > > >>>> Streaming engines.
> > > >>>>
> > > >>>> Beam is a unified Batch + Streaming processing engine from google.
> > All
> > > >> of
> > > >>>> the other Streaming engines like Flink, Spark, Apex, Storm etc...
> > are
> > > >> now
> > > >>>> vying to provide native runners that can support Beam. This kind'a
> > > >> makes
> > > >>>> Beam an abstraction over every other streaming framework.
> > > >>>>
> > > >>>> As an application developer, I would write my jobs against the
> Beam
> > > API
> > > >>>> and have the option of being able to execute the same as a Spark
> > batch
> > > >>> job,
> > > >>>> Flink streaming/batch job etc. This completely shields the
> developer
> > > >> from
> > > >>>> having to support the plethora of streaming engines out there.
> > > >>>>
> > > >>>> On the Mahout project, we built a complete logical layer for
> coding
> > up
> > > >> ML
> > > >>>> algorithms, and a physical layer that translates the logical plan
> to
> > > >> run
> > > >>> on
> > > >>>> different execution engines like Spark, Flink. There was no Beam
> > then
> > > >>> when
> > > >>>> we started that effort in early 2014.
> > > >>>> I would do it a different way today given Beam.
> > > >>>>
> > > >>>> It would be real cool by way of publicity and building a community
> > for
> > > >>>> Pirk, if Pirk were to be one of the few projects out there that
> > > support
> > > >>>> Beam.
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>>> Regards,
> > > >>>>> Chris
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <
> > t.p.ellison@gmail.com>
> > > >>>>> wrote:
> > > >>>>>
> > > >>>>>> On 17/07/16 16:57, Ellison Anne Williams wrote:
> > > >>>>>>> Suneel -- Thanks for creating the JIRA issue and pointing out
> the
> > > >>>>>> licensing
> > > >>>>>>> problems. I see that JMH is under the GNU GPL2 (
> > > >>>>>>> http://openjdk.java.net/legal/) which is not compatible with
> the
> > > >>>>> Apache
> > > >>>>>>> license (http://www.apache.org/legal/resolved.html).
> > > >>>>>>>
> > > >>>>>>> It appears that Flink just removed the benchmarking code
> instead
> > > >> of
> > > >>>>>>> re-porting it to another option.
> > > >>>>>>>
> > > >>>>>>> I would like us to port it to another license-compatible
> > > >>> benchmarking
> > > >>>>>>> framework such as Google Caliper (or something similar) instead
> > of
> > > >>>>>> removing
> > > >>>>>>> the code as the benchmarking is important for encryption
> > > >>> optimization.
> > > >>>>>>>
> > > >>>>>>> Thoughts?
> > > >>>>>>
> > > >>>>>> JMH is GPLv2 with classpath exception [1], which means that it
> > > >> cannot
> > > >>> be
> > > >>>>>> distributed as part of the ALv2 licensed works (Pirk), but there
> > is
> > > >> no
> > > >>>>>> problem with using this library as a tool / dependency at
> runtime.
> > > >>>>>> Afterall, there is no Java runtime that allows for
> redistribution
> > > >>> under
> > > >>>>>> ALv2 either!
> > > >>>>>>
> > > >>>>>> That said, running the "mvn package" target *does* put JMH
> > generated
> > > >>>>>> code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then
> begs
> > > >> the
> > > >>>>>> question why Pirk is putting test code into the library?
> > > >>>>>>
> > > >>>>>> So if the benchmark code were not part of the delivery then you
> > can
> > > >>>>>> continue to use JMH, but if that is there for a reason then we
> > would
> > > >>>>>> have to switch to a compatible licensed framework.
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>
> http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> > > >>>>>>
> > > >>>>>> Regards,
> > > >>>>>> Tim
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>
> > > >
> > >
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Suneel Marthi <su...@gmail.com>.
+1
On Sat, Jul 23, 2016 at 12:01 AM, Jacob Wilder <
jacobwilder.opensource@gmail.com> wrote:
> I'll put forward my vote that we stick with JMH and separate out any
> benchmarks that depend on it if/when we distribute binary artifacts. JMH
> seems to be better maintained (starting from the fact that it's possible to
> use it without needing to modify what you get from the source repo) and it
> seems to be easier to use.
>
> If nobody has any qualms with it I will close my pull request and we'll
> keep using JMH.
>
> —
> Jacob WIlder
>
> On Thu, Jul 21, 2016 at 7:23 AM, Tim Ellison <t....@gmail.com>
> wrote:
>
> > On 21/07/16 06:47, Suneel Marthi wrote:
> > > JMH license is not compatible with Apache and hence this thread.
> > > But if JMH is only being used for tests and if we don't package the
> tests
> > > into project artifacts on Apache mirrors, I am thinking we shuld be
> fine
> > to
> > > leave JMH as is.
> >
> > It gets a bit tricky, so here goes:
> > - the Pirk tests are Apache licensed, so obviously they are ok to
> > package and redistribute.
> >
> > - the JMH library itself is licensed as GPLv2+classpath exception.
> > This cannot be redistributed by Pirk as all Apache software must be
> > distributed under the Apache License 2.0.
> >
> > - it is ok for Pirk to have a dependency on tools (such as gpg2 for
> > artefact signing), platforms (such as Java), and /optional/ runtime
> > plug-ins that have an 'incompatible' license. These are provided by the
> > end-user, not the Pirk project.
> >
> > - where Pirk chooses to bundle the output from tools, you have to check
> > the terms that come with each tool to know if there are any restrictions.
> >
> > > I defer to Tim or Joe to make a call here.
> >
> > I see no problem with continuing to use JMH for benchmarking tests.
> >
> > Like all Apache projects, our primary distribution will be source code.
> > When it comes to creating a convenience binary artefact we will audit
> > it to check there it does not include incompatibly licensed binaries.
> >
> > HTH
> >
> > Regards,
> > Tim
> >
> > > On Wed, Jul 20, 2016 at 6:00 PM, Jacob Wilder <
> > > jacobwilder.opensource@gmail.com> wrote:
> > >
> > >> Returning to the JMH discussion, I spent a short period of time
> > integrating
> > >> Google Caliper and much more time figuring out how to run it.
> > >>
> > >> The answer is to use: `java -cp
> > >> caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
> > >> com.google.caliper.runner.CaliperMain
> > >> org.apache.pirk.benchmark.PaillierBenchmark`
> > >>
> > >> You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get
> > Caliper
> > >> to compile I had to change their pom file to bump both
> com.google.dagger
> > >> dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have only a
> > >> vague idea why this makes a difference.
> > >>
> > >> At the same time, I've looked around and it looks like we are fine to
> > >> continue using JMH (which, if I am to extrapolate off of my experience
> > in
> > >> the past two days) is easier to use. There are other Apache projects
> > that
> > >> continue to use JMH under their classpath exemption (including
> > dropwizard's
> > >> Metrics). Can we articulate a reason why we must switch and why it
> > would be
> > >> better to? I'm totally fine with cancelling this PR.
> > >>
> > >> —
> > >> Jacob WIlder
> > >>
> > >> On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <sm...@apache.org>
> > >> wrote:
> > >>
> > >>> ....and we shuld move the Beam and Streaming integration discussions
> > to
> > >> a
> > >>> different thread.
> > >>>
> > >>> On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org>
> > >>> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
> > >>> chris.harris010@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> I agree separating the benchmarking code out might be a good idea.
> > >>> Maybe
> > >>>>> we want to make this a multi-module project and the benchmarking
> be a
> > >>>>> submodule? Won't JMH still need to included via the child pom in
> the
> > >>>>> benchmarking though; I don't think it can be marked as "provided"
> or
> > >>>>> anything, right? Glad to hear if you had a different thought on how
> > to
> > >>> go
> > >>>>> with this though.
> > >>>>>
> > >>>>> Maybe it's good to create submodules for all the adapters (eg.
> Spark,
> > >>> MR,
> > >>>>> Flink, Storm etc...) too? Then we won't have just one jar needing
> to
> > >>>>> carry
> > >>>>> around all those dependencies that aren't used (and reduce
> potential
> > >>>>> version conflicts). I don't know if this would shake things up too
> > >> much
> > >>>>> to
> > >>>>> do this restructuring now, or if it is better to do now before we
> > >> move
> > >>>>> too
> > >>>>> far down the line. We'd have to look at what's Pirk "core" vs not,
> > >> but
> > >>> I
> > >>>>> don't think that's too tough to discern right now. Most of the
> > >> adapter
> > >>>>> code is in org.apache.pirk.responder.
> > >>>>>
> > >>>>
> > >>>> I have a suggestion here. Its not a good idea to be having
> submodules
> > >> for
> > >>>> every streaming engine that's out there. In the long run, its gonna
> be
> > >> a
> > >>>> maintenance and compatibility nightmare with every release of Flink,
> > >>> Spark
> > >>>> etc...
> > >>>>
> > >>>> I am guilty of having wasted the last 2 yrs of my life (and fellow
> > >>>> committers time) in adding support for Spark, Flink, H2O on the
> Apache
> > >>>> Mahout project as distributed backend engines.
> > >>>>
> > >>>> I wish now that Apache Beam was around in 2014, I would have to then
> > >> just
> > >>>> support Beam and be abstracted away from all of the other Streaming
> > >>>> frameworks.
> > >>>>
> > >>>> As a new project, I would suggest that we look into integrating with
> > >> Beam
> > >>>> and keep the codebase lean and not bloat the project with all the
> > >>> different
> > >>>> Streaming engines.
> > >>>>
> > >>>> Beam is a unified Batch + Streaming processing engine from google.
> All
> > >> of
> > >>>> the other Streaming engines like Flink, Spark, Apex, Storm etc...
> are
> > >> now
> > >>>> vying to provide native runners that can support Beam. This kind'a
> > >> makes
> > >>>> Beam an abstraction over every other streaming framework.
> > >>>>
> > >>>> As an application developer, I would write my jobs against the Beam
> > API
> > >>>> and have the option of being able to execute the same as a Spark
> batch
> > >>> job,
> > >>>> Flink streaming/batch job etc. This completely shields the developer
> > >> from
> > >>>> having to support the plethora of streaming engines out there.
> > >>>>
> > >>>> On the Mahout project, we built a complete logical layer for coding
> up
> > >> ML
> > >>>> algorithms, and a physical layer that translates the logical plan to
> > >> run
> > >>> on
> > >>>> different execution engines like Spark, Flink. There was no Beam
> then
> > >>> when
> > >>>> we started that effort in early 2014.
> > >>>> I would do it a different way today given Beam.
> > >>>>
> > >>>> It would be real cool by way of publicity and building a community
> for
> > >>>> Pirk, if Pirk were to be one of the few projects out there that
> > support
> > >>>> Beam.
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>>> Regards,
> > >>>>> Chris
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <
> t.p.ellison@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> On 17/07/16 16:57, Ellison Anne Williams wrote:
> > >>>>>>> Suneel -- Thanks for creating the JIRA issue and pointing out the
> > >>>>>> licensing
> > >>>>>>> problems. I see that JMH is under the GNU GPL2 (
> > >>>>>>> http://openjdk.java.net/legal/) which is not compatible with the
> > >>>>> Apache
> > >>>>>>> license (http://www.apache.org/legal/resolved.html).
> > >>>>>>>
> > >>>>>>> It appears that Flink just removed the benchmarking code instead
> > >> of
> > >>>>>>> re-porting it to another option.
> > >>>>>>>
> > >>>>>>> I would like us to port it to another license-compatible
> > >>> benchmarking
> > >>>>>>> framework such as Google Caliper (or something similar) instead
> of
> > >>>>>> removing
> > >>>>>>> the code as the benchmarking is important for encryption
> > >>> optimization.
> > >>>>>>>
> > >>>>>>> Thoughts?
> > >>>>>>
> > >>>>>> JMH is GPLv2 with classpath exception [1], which means that it
> > >> cannot
> > >>> be
> > >>>>>> distributed as part of the ALv2 licensed works (Pirk), but there
> is
> > >> no
> > >>>>>> problem with using this library as a tool / dependency at runtime.
> > >>>>>> Afterall, there is no Java runtime that allows for redistribution
> > >>> under
> > >>>>>> ALv2 either!
> > >>>>>>
> > >>>>>> That said, running the "mvn package" target *does* put JMH
> generated
> > >>>>>> code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs
> > >> the
> > >>>>>> question why Pirk is putting test code into the library?
> > >>>>>>
> > >>>>>> So if the benchmark code were not part of the delivery then you
> can
> > >>>>>> continue to use JMH, but if that is there for a reason then we
> would
> > >>>>>> have to switch to a compatible licensed framework.
> > >>>>>>
> > >>>>>> [1]
> > >>> http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> > >>>>>>
> > >>>>>> Regards,
> > >>>>>> Tim
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> > >
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Jacob Wilder <ja...@gmail.com>.
I'll put forward my vote that we stick with JMH and separate out any
benchmarks that depend on it if/when we distribute binary artifacts. JMH
seems to be better maintained (starting from the fact that it's possible to
use it without needing to modify what you get from the source repo) and it
seems to be easier to use.
If nobody has any qualms with it I will close my pull request and we'll
keep using JMH.
—
Jacob WIlder
On Thu, Jul 21, 2016 at 7:23 AM, Tim Ellison <t....@gmail.com> wrote:
> On 21/07/16 06:47, Suneel Marthi wrote:
> > JMH license is not compatible with Apache and hence this thread.
> > But if JMH is only being used for tests and if we don't package the tests
> > into project artifacts on Apache mirrors, I am thinking we shuld be fine
> to
> > leave JMH as is.
>
> It gets a bit tricky, so here goes:
> - the Pirk tests are Apache licensed, so obviously they are ok to
> package and redistribute.
>
> - the JMH library itself is licensed as GPLv2+classpath exception.
> This cannot be redistributed by Pirk as all Apache software must be
> distributed under the Apache License 2.0.
>
> - it is ok for Pirk to have a dependency on tools (such as gpg2 for
> artefact signing), platforms (such as Java), and /optional/ runtime
> plug-ins that have an 'incompatible' license. These are provided by the
> end-user, not the Pirk project.
>
> - where Pirk chooses to bundle the output from tools, you have to check
> the terms that come with each tool to know if there are any restrictions.
>
> > I defer to Tim or Joe to make a call here.
>
> I see no problem with continuing to use JMH for benchmarking tests.
>
> Like all Apache projects, our primary distribution will be source code.
> When it comes to creating a convenience binary artefact we will audit
> it to check there it does not include incompatibly licensed binaries.
>
> HTH
>
> Regards,
> Tim
>
> > On Wed, Jul 20, 2016 at 6:00 PM, Jacob Wilder <
> > jacobwilder.opensource@gmail.com> wrote:
> >
> >> Returning to the JMH discussion, I spent a short period of time
> integrating
> >> Google Caliper and much more time figuring out how to run it.
> >>
> >> The answer is to use: `java -cp
> >> caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
> >> com.google.caliper.runner.CaliperMain
> >> org.apache.pirk.benchmark.PaillierBenchmark`
> >>
> >> You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get
> Caliper
> >> to compile I had to change their pom file to bump both com.google.dagger
> >> dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have only a
> >> vague idea why this makes a difference.
> >>
> >> At the same time, I've looked around and it looks like we are fine to
> >> continue using JMH (which, if I am to extrapolate off of my experience
> in
> >> the past two days) is easier to use. There are other Apache projects
> that
> >> continue to use JMH under their classpath exemption (including
> dropwizard's
> >> Metrics). Can we articulate a reason why we must switch and why it
> would be
> >> better to? I'm totally fine with cancelling this PR.
> >>
> >> —
> >> Jacob WIlder
> >>
> >> On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <sm...@apache.org>
> >> wrote:
> >>
> >>> ....and we shuld move the Beam and Streaming integration discussions
> to
> >> a
> >>> different thread.
> >>>
> >>> On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org>
> >>> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
> >>> chris.harris010@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> I agree separating the benchmarking code out might be a good idea.
> >>> Maybe
> >>>>> we want to make this a multi-module project and the benchmarking be a
> >>>>> submodule? Won't JMH still need to included via the child pom in the
> >>>>> benchmarking though; I don't think it can be marked as "provided" or
> >>>>> anything, right? Glad to hear if you had a different thought on how
> to
> >>> go
> >>>>> with this though.
> >>>>>
> >>>>> Maybe it's good to create submodules for all the adapters (eg. Spark,
> >>> MR,
> >>>>> Flink, Storm etc...) too? Then we won't have just one jar needing to
> >>>>> carry
> >>>>> around all those dependencies that aren't used (and reduce potential
> >>>>> version conflicts). I don't know if this would shake things up too
> >> much
> >>>>> to
> >>>>> do this restructuring now, or if it is better to do now before we
> >> move
> >>>>> too
> >>>>> far down the line. We'd have to look at what's Pirk "core" vs not,
> >> but
> >>> I
> >>>>> don't think that's too tough to discern right now. Most of the
> >> adapter
> >>>>> code is in org.apache.pirk.responder.
> >>>>>
> >>>>
> >>>> I have a suggestion here. Its not a good idea to be having submodules
> >> for
> >>>> every streaming engine that's out there. In the long run, its gonna be
> >> a
> >>>> maintenance and compatibility nightmare with every release of Flink,
> >>> Spark
> >>>> etc...
> >>>>
> >>>> I am guilty of having wasted the last 2 yrs of my life (and fellow
> >>>> committers time) in adding support for Spark, Flink, H2O on the Apache
> >>>> Mahout project as distributed backend engines.
> >>>>
> >>>> I wish now that Apache Beam was around in 2014, I would have to then
> >> just
> >>>> support Beam and be abstracted away from all of the other Streaming
> >>>> frameworks.
> >>>>
> >>>> As a new project, I would suggest that we look into integrating with
> >> Beam
> >>>> and keep the codebase lean and not bloat the project with all the
> >>> different
> >>>> Streaming engines.
> >>>>
> >>>> Beam is a unified Batch + Streaming processing engine from google. All
> >> of
> >>>> the other Streaming engines like Flink, Spark, Apex, Storm etc... are
> >> now
> >>>> vying to provide native runners that can support Beam. This kind'a
> >> makes
> >>>> Beam an abstraction over every other streaming framework.
> >>>>
> >>>> As an application developer, I would write my jobs against the Beam
> API
> >>>> and have the option of being able to execute the same as a Spark batch
> >>> job,
> >>>> Flink streaming/batch job etc. This completely shields the developer
> >> from
> >>>> having to support the plethora of streaming engines out there.
> >>>>
> >>>> On the Mahout project, we built a complete logical layer for coding up
> >> ML
> >>>> algorithms, and a physical layer that translates the logical plan to
> >> run
> >>> on
> >>>> different execution engines like Spark, Flink. There was no Beam then
> >>> when
> >>>> we started that effort in early 2014.
> >>>> I would do it a different way today given Beam.
> >>>>
> >>>> It would be real cool by way of publicity and building a community for
> >>>> Pirk, if Pirk were to be one of the few projects out there that
> support
> >>>> Beam.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>> Regards,
> >>>>> Chris
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> On 17/07/16 16:57, Ellison Anne Williams wrote:
> >>>>>>> Suneel -- Thanks for creating the JIRA issue and pointing out the
> >>>>>> licensing
> >>>>>>> problems. I see that JMH is under the GNU GPL2 (
> >>>>>>> http://openjdk.java.net/legal/) which is not compatible with the
> >>>>> Apache
> >>>>>>> license (http://www.apache.org/legal/resolved.html).
> >>>>>>>
> >>>>>>> It appears that Flink just removed the benchmarking code instead
> >> of
> >>>>>>> re-porting it to another option.
> >>>>>>>
> >>>>>>> I would like us to port it to another license-compatible
> >>> benchmarking
> >>>>>>> framework such as Google Caliper (or something similar) instead of
> >>>>>> removing
> >>>>>>> the code as the benchmarking is important for encryption
> >>> optimization.
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>
> >>>>>> JMH is GPLv2 with classpath exception [1], which means that it
> >> cannot
> >>> be
> >>>>>> distributed as part of the ALv2 licensed works (Pirk), but there is
> >> no
> >>>>>> problem with using this library as a tool / dependency at runtime.
> >>>>>> Afterall, there is no Java runtime that allows for redistribution
> >>> under
> >>>>>> ALv2 either!
> >>>>>>
> >>>>>> That said, running the "mvn package" target *does* put JMH generated
> >>>>>> code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs
> >> the
> >>>>>> question why Pirk is putting test code into the library?
> >>>>>>
> >>>>>> So if the benchmark code were not part of the delivery then you can
> >>>>>> continue to use JMH, but if that is there for a reason then we would
> >>>>>> have to switch to a compatible licensed framework.
> >>>>>>
> >>>>>> [1]
> >>> http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> >>>>>>
> >>>>>> Regards,
> >>>>>> Tim
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Tim Ellison <t....@gmail.com>.
On 21/07/16 06:47, Suneel Marthi wrote:
> JMH license is not compatible with Apache and hence this thread.
> But if JMH is only being used for tests and if we don't package the tests
> into project artifacts on Apache mirrors, I am thinking we shuld be fine to
> leave JMH as is.
It gets a bit tricky, so here goes:
- the Pirk tests are Apache licensed, so obviously they are ok to
package and redistribute.
- the JMH library itself is licensed as GPLv2+classpath exception.
This cannot be redistributed by Pirk as all Apache software must be
distributed under the Apache License 2.0.
- it is ok for Pirk to have a dependency on tools (such as gpg2 for
artefact signing), platforms (such as Java), and /optional/ runtime
plug-ins that have an 'incompatible' license. These are provided by the
end-user, not the Pirk project.
- where Pirk chooses to bundle the output from tools, you have to check
the terms that come with each tool to know if there are any restrictions.
> I defer to Tim or Joe to make a call here.
I see no problem with continuing to use JMH for benchmarking tests.
Like all Apache projects, our primary distribution will be source code.
When it comes to creating a convenience binary artefact we will audit
it to check there it does not include incompatibly licensed binaries.
HTH
Regards,
Tim
> On Wed, Jul 20, 2016 at 6:00 PM, Jacob Wilder <
> jacobwilder.opensource@gmail.com> wrote:
>
>> Returning to the JMH discussion, I spent a short period of time integrating
>> Google Caliper and much more time figuring out how to run it.
>>
>> The answer is to use: `java -cp
>> caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
>> com.google.caliper.runner.CaliperMain
>> org.apache.pirk.benchmark.PaillierBenchmark`
>>
>> You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get Caliper
>> to compile I had to change their pom file to bump both com.google.dagger
>> dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have only a
>> vague idea why this makes a difference.
>>
>> At the same time, I've looked around and it looks like we are fine to
>> continue using JMH (which, if I am to extrapolate off of my experience in
>> the past two days) is easier to use. There are other Apache projects that
>> continue to use JMH under their classpath exemption (including dropwizard's
>> Metrics). Can we articulate a reason why we must switch and why it would be
>> better to? I'm totally fine with cancelling this PR.
>>
>> \u2014
>> Jacob WIlder
>>
>> On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <sm...@apache.org>
>> wrote:
>>
>>> ....and we shuld move the Beam and Streaming integration discussions to
>> a
>>> different thread.
>>>
>>> On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
>>> chris.harris010@gmail.com>
>>>> wrote:
>>>>
>>>>> I agree separating the benchmarking code out might be a good idea.
>>> Maybe
>>>>> we want to make this a multi-module project and the benchmarking be a
>>>>> submodule? Won't JMH still need to included via the child pom in the
>>>>> benchmarking though; I don't think it can be marked as "provided" or
>>>>> anything, right? Glad to hear if you had a different thought on how to
>>> go
>>>>> with this though.
>>>>>
>>>>> Maybe it's good to create submodules for all the adapters (eg. Spark,
>>> MR,
>>>>> Flink, Storm etc...) too? Then we won't have just one jar needing to
>>>>> carry
>>>>> around all those dependencies that aren't used (and reduce potential
>>>>> version conflicts). I don't know if this would shake things up too
>> much
>>>>> to
>>>>> do this restructuring now, or if it is better to do now before we
>> move
>>>>> too
>>>>> far down the line. We'd have to look at what's Pirk "core" vs not,
>> but
>>> I
>>>>> don't think that's too tough to discern right now. Most of the
>> adapter
>>>>> code is in org.apache.pirk.responder.
>>>>>
>>>>
>>>> I have a suggestion here. Its not a good idea to be having submodules
>> for
>>>> every streaming engine that's out there. In the long run, its gonna be
>> a
>>>> maintenance and compatibility nightmare with every release of Flink,
>>> Spark
>>>> etc...
>>>>
>>>> I am guilty of having wasted the last 2 yrs of my life (and fellow
>>>> committers time) in adding support for Spark, Flink, H2O on the Apache
>>>> Mahout project as distributed backend engines.
>>>>
>>>> I wish now that Apache Beam was around in 2014, I would have to then
>> just
>>>> support Beam and be abstracted away from all of the other Streaming
>>>> frameworks.
>>>>
>>>> As a new project, I would suggest that we look into integrating with
>> Beam
>>>> and keep the codebase lean and not bloat the project with all the
>>> different
>>>> Streaming engines.
>>>>
>>>> Beam is a unified Batch + Streaming processing engine from google. All
>> of
>>>> the other Streaming engines like Flink, Spark, Apex, Storm etc... are
>> now
>>>> vying to provide native runners that can support Beam. This kind'a
>> makes
>>>> Beam an abstraction over every other streaming framework.
>>>>
>>>> As an application developer, I would write my jobs against the Beam API
>>>> and have the option of being able to execute the same as a Spark batch
>>> job,
>>>> Flink streaming/batch job etc. This completely shields the developer
>> from
>>>> having to support the plethora of streaming engines out there.
>>>>
>>>> On the Mahout project, we built a complete logical layer for coding up
>> ML
>>>> algorithms, and a physical layer that translates the logical plan to
>> run
>>> on
>>>> different execution engines like Spark, Flink. There was no Beam then
>>> when
>>>> we started that effort in early 2014.
>>>> I would do it a different way today given Beam.
>>>>
>>>> It would be real cool by way of publicity and building a community for
>>>> Pirk, if Pirk were to be one of the few projects out there that support
>>>> Beam.
>>>>
>>>>
>>>>
>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On 17/07/16 16:57, Ellison Anne Williams wrote:
>>>>>>> Suneel -- Thanks for creating the JIRA issue and pointing out the
>>>>>> licensing
>>>>>>> problems. I see that JMH is under the GNU GPL2 (
>>>>>>> http://openjdk.java.net/legal/) which is not compatible with the
>>>>> Apache
>>>>>>> license (http://www.apache.org/legal/resolved.html).
>>>>>>>
>>>>>>> It appears that Flink just removed the benchmarking code instead
>> of
>>>>>>> re-porting it to another option.
>>>>>>>
>>>>>>> I would like us to port it to another license-compatible
>>> benchmarking
>>>>>>> framework such as Google Caliper (or something similar) instead of
>>>>>> removing
>>>>>>> the code as the benchmarking is important for encryption
>>> optimization.
>>>>>>>
>>>>>>> Thoughts?
>>>>>>
>>>>>> JMH is GPLv2 with classpath exception [1], which means that it
>> cannot
>>> be
>>>>>> distributed as part of the ALv2 licensed works (Pirk), but there is
>> no
>>>>>> problem with using this library as a tool / dependency at runtime.
>>>>>> Afterall, there is no Java runtime that allows for redistribution
>>> under
>>>>>> ALv2 either!
>>>>>>
>>>>>> That said, running the "mvn package" target *does* put JMH generated
>>>>>> code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs
>> the
>>>>>> question why Pirk is putting test code into the library?
>>>>>>
>>>>>> So if the benchmark code were not part of the delivery then you can
>>>>>> continue to use JMH, but if that is there for a reason then we would
>>>>>> have to switch to a compatible licensed framework.
>>>>>>
>>>>>> [1]
>>> http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
>>>>>>
>>>>>> Regards,
>>>>>> Tim
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: PIRK-7 -- JMH license issues
Posted by Suneel Marthi <sm...@apache.org>.
JMH license is not compatible with Apache and hence this thread.
But if JMH is only being used for tests and if we don't package the tests
into project artifacts on Apache mirrors, I am thinking we shuld be fine to
leave JMH as is.
I defer to Tim or Joe to make a call here.
On Wed, Jul 20, 2016 at 6:00 PM, Jacob Wilder <
jacobwilder.opensource@gmail.com> wrote:
> Returning to the JMH discussion, I spent a short period of time integrating
> Google Caliper and much more time figuring out how to run it.
>
> The answer is to use: `java -cp
> caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
> com.google.caliper.runner.CaliperMain
> org.apache.pirk.benchmark.PaillierBenchmark`
>
> You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get Caliper
> to compile I had to change their pom file to bump both com.google.dagger
> dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have only a
> vague idea why this makes a difference.
>
> At the same time, I've looked around and it looks like we are fine to
> continue using JMH (which, if I am to extrapolate off of my experience in
> the past two days) is easier to use. There are other Apache projects that
> continue to use JMH under their classpath exemption (including dropwizard's
> Metrics). Can we articulate a reason why we must switch and why it would be
> better to? I'm totally fine with cancelling this PR.
>
> —
> Jacob WIlder
>
> On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <sm...@apache.org>
> wrote:
>
> > ....and we shuld move the Beam and Streaming integration discussions to
> a
> > different thread.
> >
> > On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org>
> > wrote:
> >
> > >
> > >
> > > On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
> > chris.harris010@gmail.com>
> > > wrote:
> > >
> > >> I agree separating the benchmarking code out might be a good idea.
> > Maybe
> > >> we want to make this a multi-module project and the benchmarking be a
> > >> submodule? Won't JMH still need to included via the child pom in the
> > >> benchmarking though; I don't think it can be marked as "provided" or
> > >> anything, right? Glad to hear if you had a different thought on how to
> > go
> > >> with this though.
> > >>
> > >> Maybe it's good to create submodules for all the adapters (eg. Spark,
> > MR,
> > >> Flink, Storm etc...) too? Then we won't have just one jar needing to
> > >> carry
> > >> around all those dependencies that aren't used (and reduce potential
> > >> version conflicts). I don't know if this would shake things up too
> much
> > >> to
> > >> do this restructuring now, or if it is better to do now before we
> move
> > >> too
> > >> far down the line. We'd have to look at what's Pirk "core" vs not,
> but
> > I
> > >> don't think that's too tough to discern right now. Most of the
> adapter
> > >> code is in org.apache.pirk.responder.
> > >>
> > >
> > > I have a suggestion here. Its not a good idea to be having submodules
> for
> > > every streaming engine that's out there. In the long run, its gonna be
> a
> > > maintenance and compatibility nightmare with every release of Flink,
> > Spark
> > > etc...
> > >
> > > I am guilty of having wasted the last 2 yrs of my life (and fellow
> > > committers time) in adding support for Spark, Flink, H2O on the Apache
> > > Mahout project as distributed backend engines.
> > >
> > > I wish now that Apache Beam was around in 2014, I would have to then
> just
> > > support Beam and be abstracted away from all of the other Streaming
> > > frameworks.
> > >
> > > As a new project, I would suggest that we look into integrating with
> Beam
> > > and keep the codebase lean and not bloat the project with all the
> > different
> > > Streaming engines.
> > >
> > > Beam is a unified Batch + Streaming processing engine from google. All
> of
> > > the other Streaming engines like Flink, Spark, Apex, Storm etc... are
> now
> > > vying to provide native runners that can support Beam. This kind'a
> makes
> > > Beam an abstraction over every other streaming framework.
> > >
> > > As an application developer, I would write my jobs against the Beam API
> > > and have the option of being able to execute the same as a Spark batch
> > job,
> > > Flink streaming/batch job etc. This completely shields the developer
> from
> > > having to support the plethora of streaming engines out there.
> > >
> > > On the Mahout project, we built a complete logical layer for coding up
> ML
> > > algorithms, and a physical layer that translates the logical plan to
> run
> > on
> > > different execution engines like Spark, Flink. There was no Beam then
> > when
> > > we started that effort in early 2014.
> > > I would do it a different way today given Beam.
> > >
> > > It would be real cool by way of publicity and building a community for
> > > Pirk, if Pirk were to be one of the few projects out there that support
> > > Beam.
> > >
> > >
> > >
> > >
> > >> Regards,
> > >> Chris
> > >>
> > >>
> > >>
> > >> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
> > >> wrote:
> > >>
> > >> > On 17/07/16 16:57, Ellison Anne Williams wrote:
> > >> > > Suneel -- Thanks for creating the JIRA issue and pointing out the
> > >> > licensing
> > >> > > problems. I see that JMH is under the GNU GPL2 (
> > >> > > http://openjdk.java.net/legal/) which is not compatible with the
> > >> Apache
> > >> > > license (http://www.apache.org/legal/resolved.html).
> > >> > >
> > >> > > It appears that Flink just removed the benchmarking code instead
> of
> > >> > > re-porting it to another option.
> > >> > >
> > >> > > I would like us to port it to another license-compatible
> > benchmarking
> > >> > > framework such as Google Caliper (or something similar) instead of
> > >> > removing
> > >> > > the code as the benchmarking is important for encryption
> > optimization.
> > >> > >
> > >> > > Thoughts?
> > >> >
> > >> > JMH is GPLv2 with classpath exception [1], which means that it
> cannot
> > be
> > >> > distributed as part of the ALv2 licensed works (Pirk), but there is
> no
> > >> > problem with using this library as a tool / dependency at runtime.
> > >> > Afterall, there is no Java runtime that allows for redistribution
> > under
> > >> > ALv2 either!
> > >> >
> > >> > That said, running the "mvn package" target *does* put JMH generated
> > >> > code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs
> the
> > >> > question why Pirk is putting test code into the library?
> > >> >
> > >> > So if the benchmark code were not part of the delivery then you can
> > >> > continue to use JMH, but if that is there for a reason then we would
> > >> > have to switch to a compatible licensed framework.
> > >> >
> > >> > [1]
> > http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> > >> >
> > >> > Regards,
> > >> > Tim
> > >> >
> > >>
> > >
> > >
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Jacob Wilder <ja...@gmail.com>.
Returning to the JMH discussion, I spent a short period of time integrating
Google Caliper and much more time figuring out how to run it.
The answer is to use: `java -cp
caliper-1.0-SNAPSHOT-all.jar:pirk-0.0.1-SNAPSHOT-exe.jar:.
com.google.caliper.runner.CaliperMain
org.apache.pirk.benchmark.PaillierBenchmark`
You need to provide your own caliper-1.0-SNAPSHOT-all.jar. To get Caliper
to compile I had to change their pom file to bump both com.google.dagger
dependencies from 2.1-SNAPSHOT to 2.4. I don't like that I have only a
vague idea why this makes a difference.
At the same time, I've looked around and it looks like we are fine to
continue using JMH (which, if I am to extrapolate off of my experience in
the past two days) is easier to use. There are other Apache projects that
continue to use JMH under their classpath exemption (including dropwizard's
Metrics). Can we articulate a reason why we must switch and why it would be
better to? I'm totally fine with cancelling this PR.
—
Jacob WIlder
On Mon, Jul 18, 2016 at 11:34 AM, Suneel Marthi <sm...@apache.org> wrote:
> ....and we shuld move the Beam and Streaming integration discussions to a
> different thread.
>
> On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org>
> wrote:
>
> >
> >
> > On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <
> chris.harris010@gmail.com>
> > wrote:
> >
> >> I agree separating the benchmarking code out might be a good idea.
> Maybe
> >> we want to make this a multi-module project and the benchmarking be a
> >> submodule? Won't JMH still need to included via the child pom in the
> >> benchmarking though; I don't think it can be marked as "provided" or
> >> anything, right? Glad to hear if you had a different thought on how to
> go
> >> with this though.
> >>
> >> Maybe it's good to create submodules for all the adapters (eg. Spark,
> MR,
> >> Flink, Storm etc...) too? Then we won't have just one jar needing to
> >> carry
> >> around all those dependencies that aren't used (and reduce potential
> >> version conflicts). I don't know if this would shake things up too much
> >> to
> >> do this restructuring now, or if it is better to do now before we move
> >> too
> >> far down the line. We'd have to look at what's Pirk "core" vs not, but
> I
> >> don't think that's too tough to discern right now. Most of the adapter
> >> code is in org.apache.pirk.responder.
> >>
> >
> > I have a suggestion here. Its not a good idea to be having submodules for
> > every streaming engine that's out there. In the long run, its gonna be a
> > maintenance and compatibility nightmare with every release of Flink,
> Spark
> > etc...
> >
> > I am guilty of having wasted the last 2 yrs of my life (and fellow
> > committers time) in adding support for Spark, Flink, H2O on the Apache
> > Mahout project as distributed backend engines.
> >
> > I wish now that Apache Beam was around in 2014, I would have to then just
> > support Beam and be abstracted away from all of the other Streaming
> > frameworks.
> >
> > As a new project, I would suggest that we look into integrating with Beam
> > and keep the codebase lean and not bloat the project with all the
> different
> > Streaming engines.
> >
> > Beam is a unified Batch + Streaming processing engine from google. All of
> > the other Streaming engines like Flink, Spark, Apex, Storm etc... are now
> > vying to provide native runners that can support Beam. This kind'a makes
> > Beam an abstraction over every other streaming framework.
> >
> > As an application developer, I would write my jobs against the Beam API
> > and have the option of being able to execute the same as a Spark batch
> job,
> > Flink streaming/batch job etc. This completely shields the developer from
> > having to support the plethora of streaming engines out there.
> >
> > On the Mahout project, we built a complete logical layer for coding up ML
> > algorithms, and a physical layer that translates the logical plan to run
> on
> > different execution engines like Spark, Flink. There was no Beam then
> when
> > we started that effort in early 2014.
> > I would do it a different way today given Beam.
> >
> > It would be real cool by way of publicity and building a community for
> > Pirk, if Pirk were to be one of the few projects out there that support
> > Beam.
> >
> >
> >
> >
> >> Regards,
> >> Chris
> >>
> >>
> >>
> >> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
> >> wrote:
> >>
> >> > On 17/07/16 16:57, Ellison Anne Williams wrote:
> >> > > Suneel -- Thanks for creating the JIRA issue and pointing out the
> >> > licensing
> >> > > problems. I see that JMH is under the GNU GPL2 (
> >> > > http://openjdk.java.net/legal/) which is not compatible with the
> >> Apache
> >> > > license (http://www.apache.org/legal/resolved.html).
> >> > >
> >> > > It appears that Flink just removed the benchmarking code instead of
> >> > > re-porting it to another option.
> >> > >
> >> > > I would like us to port it to another license-compatible
> benchmarking
> >> > > framework such as Google Caliper (or something similar) instead of
> >> > removing
> >> > > the code as the benchmarking is important for encryption
> optimization.
> >> > >
> >> > > Thoughts?
> >> >
> >> > JMH is GPLv2 with classpath exception [1], which means that it cannot
> be
> >> > distributed as part of the ALv2 licensed works (Pirk), but there is no
> >> > problem with using this library as a tool / dependency at runtime.
> >> > Afterall, there is no Java runtime that allows for redistribution
> under
> >> > ALv2 either!
> >> >
> >> > That said, running the "mvn package" target *does* put JMH generated
> >> > code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs the
> >> > question why Pirk is putting test code into the library?
> >> >
> >> > So if the benchmark code were not part of the delivery then you can
> >> > continue to use JMH, but if that is there for a reason then we would
> >> > have to switch to a compatible licensed framework.
> >> >
> >> > [1]
> http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> >> >
> >> > Regards,
> >> > Tim
> >> >
> >>
> >
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Suneel Marthi <sm...@apache.org>.
....and we shuld move the Beam and Streaming integration discussions to a
different thread.
On Mon, Jul 18, 2016 at 11:32 AM, Suneel Marthi <sm...@apache.org> wrote:
>
>
> On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <ch...@gmail.com>
> wrote:
>
>> I agree separating the benchmarking code out might be a good idea. Maybe
>> we want to make this a multi-module project and the benchmarking be a
>> submodule? Won't JMH still need to included via the child pom in the
>> benchmarking though; I don't think it can be marked as "provided" or
>> anything, right? Glad to hear if you had a different thought on how to go
>> with this though.
>>
>> Maybe it's good to create submodules for all the adapters (eg. Spark, MR,
>> Flink, Storm etc...) too? Then we won't have just one jar needing to
>> carry
>> around all those dependencies that aren't used (and reduce potential
>> version conflicts). I don't know if this would shake things up too much
>> to
>> do this restructuring now, or if it is better to do now before we move
>> too
>> far down the line. We'd have to look at what's Pirk "core" vs not, but I
>> don't think that's too tough to discern right now. Most of the adapter
>> code is in org.apache.pirk.responder.
>>
>
> I have a suggestion here. Its not a good idea to be having submodules for
> every streaming engine that's out there. In the long run, its gonna be a
> maintenance and compatibility nightmare with every release of Flink, Spark
> etc...
>
> I am guilty of having wasted the last 2 yrs of my life (and fellow
> committers time) in adding support for Spark, Flink, H2O on the Apache
> Mahout project as distributed backend engines.
>
> I wish now that Apache Beam was around in 2014, I would have to then just
> support Beam and be abstracted away from all of the other Streaming
> frameworks.
>
> As a new project, I would suggest that we look into integrating with Beam
> and keep the codebase lean and not bloat the project with all the different
> Streaming engines.
>
> Beam is a unified Batch + Streaming processing engine from google. All of
> the other Streaming engines like Flink, Spark, Apex, Storm etc... are now
> vying to provide native runners that can support Beam. This kind'a makes
> Beam an abstraction over every other streaming framework.
>
> As an application developer, I would write my jobs against the Beam API
> and have the option of being able to execute the same as a Spark batch job,
> Flink streaming/batch job etc. This completely shields the developer from
> having to support the plethora of streaming engines out there.
>
> On the Mahout project, we built a complete logical layer for coding up ML
> algorithms, and a physical layer that translates the logical plan to run on
> different execution engines like Spark, Flink. There was no Beam then when
> we started that effort in early 2014.
> I would do it a different way today given Beam.
>
> It would be real cool by way of publicity and building a community for
> Pirk, if Pirk were to be one of the few projects out there that support
> Beam.
>
>
>
>
>> Regards,
>> Chris
>>
>>
>>
>> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
>> wrote:
>>
>> > On 17/07/16 16:57, Ellison Anne Williams wrote:
>> > > Suneel -- Thanks for creating the JIRA issue and pointing out the
>> > licensing
>> > > problems. I see that JMH is under the GNU GPL2 (
>> > > http://openjdk.java.net/legal/) which is not compatible with the
>> Apache
>> > > license (http://www.apache.org/legal/resolved.html).
>> > >
>> > > It appears that Flink just removed the benchmarking code instead of
>> > > re-porting it to another option.
>> > >
>> > > I would like us to port it to another license-compatible benchmarking
>> > > framework such as Google Caliper (or something similar) instead of
>> > removing
>> > > the code as the benchmarking is important for encryption optimization.
>> > >
>> > > Thoughts?
>> >
>> > JMH is GPLv2 with classpath exception [1], which means that it cannot be
>> > distributed as part of the ALv2 licensed works (Pirk), but there is no
>> > problem with using this library as a tool / dependency at runtime.
>> > Afterall, there is no Java runtime that allows for redistribution under
>> > ALv2 either!
>> >
>> > That said, running the "mvn package" target *does* put JMH generated
>> > code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs the
>> > question why Pirk is putting test code into the library?
>> >
>> > So if the benchmark code were not part of the delivery then you can
>> > continue to use JMH, but if that is there for a reason then we would
>> > have to switch to a compatible licensed framework.
>> >
>> > [1] http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
>> >
>> > Regards,
>> > Tim
>> >
>>
>
>
Re: PIRK-7 -- JMH license issues
Posted by Suneel Marthi <sm...@apache.org>.
On Mon, Jul 18, 2016 at 11:03 AM, Chris Harris <ch...@gmail.com>
wrote:
> I agree separating the benchmarking code out might be a good idea. Maybe
> we want to make this a multi-module project and the benchmarking be a
> submodule? Won't JMH still need to included via the child pom in the
> benchmarking though; I don't think it can be marked as "provided" or
> anything, right? Glad to hear if you had a different thought on how to go
> with this though.
>
> Maybe it's good to create submodules for all the adapters (eg. Spark, MR,
> Flink, Storm etc...) too? Then we won't have just one jar needing to carry
> around all those dependencies that aren't used (and reduce potential
> version conflicts). I don't know if this would shake things up too much to
> do this restructuring now, or if it is better to do now before we move too
> far down the line. We'd have to look at what's Pirk "core" vs not, but I
> don't think that's too tough to discern right now. Most of the adapter
> code is in org.apache.pirk.responder.
>
I have a suggestion here. Its not a good idea to be having submodules for
every streaming engine that's out there. In the long run, its gonna be a
maintenance and compatibility nightmare with every release of Flink, Spark
etc...
I am guilty of having wasted the last 2 yrs of my life (and fellow
committers time) in adding support for Spark, Flink, H2O on the Apache
Mahout project as distributed backend engines.
I wish now that Apache Beam was around in 2014, I would have to then just
support Beam and be abstracted away from all of the other Streaming
frameworks.
As a new project, I would suggest that we look into integrating with Beam
and keep the codebase lean and not bloat the project with all the different
Streaming engines.
Beam is a unified Batch + Streaming processing engine from google. All of
the other Streaming engines like Flink, Spark, Apex, Storm etc... are now
vying to provide native runners that can support Beam. This kind'a makes
Beam an abstraction over every other streaming framework.
As an application developer, I would write my jobs against the Beam API and
have the option of being able to execute the same as a Spark batch job,
Flink streaming/batch job etc. This completely shields the developer from
having to support the plethora of streaming engines out there.
On the Mahout project, we built a complete logical layer for coding up ML
algorithms, and a physical layer that translates the logical plan to run on
different execution engines like Spark, Flink. There was no Beam then when
we started that effort in early 2014.
I would do it a different way today given Beam.
It would be real cool by way of publicity and building a community for
Pirk, if Pirk were to be one of the few projects out there that support
Beam.
> Regards,
> Chris
>
>
>
> On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com>
> wrote:
>
> > On 17/07/16 16:57, Ellison Anne Williams wrote:
> > > Suneel -- Thanks for creating the JIRA issue and pointing out the
> > licensing
> > > problems. I see that JMH is under the GNU GPL2 (
> > > http://openjdk.java.net/legal/) which is not compatible with the
> Apache
> > > license (http://www.apache.org/legal/resolved.html).
> > >
> > > It appears that Flink just removed the benchmarking code instead of
> > > re-porting it to another option.
> > >
> > > I would like us to port it to another license-compatible benchmarking
> > > framework such as Google Caliper (or something similar) instead of
> > removing
> > > the code as the benchmarking is important for encryption optimization.
> > >
> > > Thoughts?
> >
> > JMH is GPLv2 with classpath exception [1], which means that it cannot be
> > distributed as part of the ALv2 licensed works (Pirk), but there is no
> > problem with using this library as a tool / dependency at runtime.
> > Afterall, there is no Java runtime that allows for redistribution under
> > ALv2 either!
> >
> > That said, running the "mvn package" target *does* put JMH generated
> > code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs the
> > question why Pirk is putting test code into the library?
> >
> > So if the benchmark code were not part of the delivery then you can
> > continue to use JMH, but if that is there for a reason then we would
> > have to switch to a compatible licensed framework.
> >
> > [1] http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
> >
> > Regards,
> > Tim
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Chris Harris <ch...@gmail.com>.
I agree separating the benchmarking code out might be a good idea. Maybe
we want to make this a multi-module project and the benchmarking be a
submodule? Won't JMH still need to included via the child pom in the
benchmarking though; I don't think it can be marked as "provided" or
anything, right? Glad to hear if you had a different thought on how to go
with this though.
Maybe it's good to create submodules for all the adapters (eg. Spark, MR,
Flink, Storm etc...) too? Then we won't have just one jar needing to carry
around all those dependencies that aren't used (and reduce potential
version conflicts). I don't know if this would shake things up too much to
do this restructuring now, or if it is better to do now before we move too
far down the line. We'd have to look at what's Pirk "core" vs not, but I
don't think that's too tough to discern right now. Most of the adapter
code is in org.apache.pirk.responder.
Regards,
Chris
On Mon, Jul 18, 2016 at 6:25 AM, Tim Ellison <t....@gmail.com> wrote:
> On 17/07/16 16:57, Ellison Anne Williams wrote:
> > Suneel -- Thanks for creating the JIRA issue and pointing out the
> licensing
> > problems. I see that JMH is under the GNU GPL2 (
> > http://openjdk.java.net/legal/) which is not compatible with the Apache
> > license (http://www.apache.org/legal/resolved.html).
> >
> > It appears that Flink just removed the benchmarking code instead of
> > re-porting it to another option.
> >
> > I would like us to port it to another license-compatible benchmarking
> > framework such as Google Caliper (or something similar) instead of
> removing
> > the code as the benchmarking is important for encryption optimization.
> >
> > Thoughts?
>
> JMH is GPLv2 with classpath exception [1], which means that it cannot be
> distributed as part of the ALv2 licensed works (Pirk), but there is no
> problem with using this library as a tool / dependency at runtime.
> Afterall, there is no Java runtime that allows for redistribution under
> ALv2 either!
>
> That said, running the "mvn package" target *does* put JMH generated
> code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs the
> question why Pirk is putting test code into the library?
>
> So if the benchmark code were not part of the delivery then you can
> continue to use JMH, but if that is there for a reason then we would
> have to switch to a compatible licensed framework.
>
> [1] http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
>
> Regards,
> Tim
>
Re: PIRK-7 -- JMH license issues
Posted by Tim Ellison <t....@gmail.com>.
On 17/07/16 16:57, Ellison Anne Williams wrote:
> Suneel -- Thanks for creating the JIRA issue and pointing out the licensing
> problems. I see that JMH is under the GNU GPL2 (
> http://openjdk.java.net/legal/) which is not compatible with the Apache
> license (http://www.apache.org/legal/resolved.html).
>
> It appears that Flink just removed the benchmarking code instead of
> re-porting it to another option.
>
> I would like us to port it to another license-compatible benchmarking
> framework such as Google Caliper (or something similar) instead of removing
> the code as the benchmarking is important for encryption optimization.
>
> Thoughts?
JMH is GPLv2 with classpath exception [1], which means that it cannot be
distributed as part of the ALv2 licensed works (Pirk), but there is no
problem with using this library as a tool / dependency at runtime.
Afterall, there is no Java runtime that allows for redistribution under
ALv2 either!
That said, running the "mvn package" target *does* put JMH generated
code into the resulting pirk-0.0.1-SNAPSHOT.jar -- which then begs the
question why Pirk is putting test code into the library?
So if the benchmark code were not part of the delivery then you can
continue to use JMH, but if that is there for a reason then we would
have to switch to a compatible licensed framework.
[1] http://hg.openjdk.java.net/code-tools/jmh/file/c050a47b2b37/LICENSE
Regards,
Tim
Re: PIRK-7 -- JMH license issues
Posted by Jacob Wilder <me...@jacobwilder.org>.
I wrote the original benchmarks so I'll take care of migrating us to Google
Caliper (unless someone beats me to the punch). Caliper seems to be similar
enough that we don't have to scrap overall benchmark design.
—
Jacob Wilder
On Sun, Jul 17, 2016 at 12:09 PM, Ellison Anne Williams <
eawilliamspirk@gmail.com> wrote:
> Ok - let's take a close look at Google Caliper (unless anyone has a better
> option) and see if we can go ahead and port
>
> On Sun, Jul 17, 2016 at 12:03 PM, Suneel Marthi <sm...@apache.org>
> wrote:
>
> > I completely agree that we should port that code to Google Caliper.
> >
> > The benchmarking code wasn't a high priority for Flink project and hence
> > they chose to just remove the code.
> >
> > The alternatives to JMH are - Google Caliper and Metrics (
> > http://metrics.dropwizard.io/3.1.0/) and anything else folks are aware
> of
> > ??
> >
> > My personal preference would be Google Caliper, we had used that in the
> > past to micro-benchmark Mahout's legacy Math Linear Algebra backend.
> >
> > Suneel
> >
> >
> >
> > On Sun, Jul 17, 2016 at 11:57 AM, Ellison Anne Williams <
> > eawilliamspirk@gmail.com> wrote:
> >
> > > Suneel -- Thanks for creating the JIRA issue and pointing out the
> > licensing
> > > problems. I see that JMH is under the GNU GPL2 (
> > > http://openjdk.java.net/legal/) which is not compatible with the
> Apache
> > > license (http://www.apache.org/legal/resolved.html).
> > >
> > > It appears that Flink just removed the benchmarking code instead of
> > > re-porting it to another option.
> > >
> > > I would like us to port it to another license-compatible benchmarking
> > > framework such as Google Caliper (or something similar) instead of
> > removing
> > > the code as the benchmarking is important for encryption optimization.
> > >
> > > Thoughts?
> > >
> > > Thanks!
> > >
> > > Ellison Anne
> > >
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Ellison Anne Williams <ea...@gmail.com>.
Ok - let's take a close look at Google Caliper (unless anyone has a better
option) and see if we can go ahead and port
On Sun, Jul 17, 2016 at 12:03 PM, Suneel Marthi <sm...@apache.org> wrote:
> I completely agree that we should port that code to Google Caliper.
>
> The benchmarking code wasn't a high priority for Flink project and hence
> they chose to just remove the code.
>
> The alternatives to JMH are - Google Caliper and Metrics (
> http://metrics.dropwizard.io/3.1.0/) and anything else folks are aware of
> ??
>
> My personal preference would be Google Caliper, we had used that in the
> past to micro-benchmark Mahout's legacy Math Linear Algebra backend.
>
> Suneel
>
>
>
> On Sun, Jul 17, 2016 at 11:57 AM, Ellison Anne Williams <
> eawilliamspirk@gmail.com> wrote:
>
> > Suneel -- Thanks for creating the JIRA issue and pointing out the
> licensing
> > problems. I see that JMH is under the GNU GPL2 (
> > http://openjdk.java.net/legal/) which is not compatible with the Apache
> > license (http://www.apache.org/legal/resolved.html).
> >
> > It appears that Flink just removed the benchmarking code instead of
> > re-porting it to another option.
> >
> > I would like us to port it to another license-compatible benchmarking
> > framework such as Google Caliper (or something similar) instead of
> removing
> > the code as the benchmarking is important for encryption optimization.
> >
> > Thoughts?
> >
> > Thanks!
> >
> > Ellison Anne
> >
>
Re: PIRK-7 -- JMH license issues
Posted by Suneel Marthi <sm...@apache.org>.
I completely agree that we should port that code to Google Caliper.
The benchmarking code wasn't a high priority for Flink project and hence
they chose to just remove the code.
The alternatives to JMH are - Google Caliper and Metrics (
http://metrics.dropwizard.io/3.1.0/) and anything else folks are aware of ??
My personal preference would be Google Caliper, we had used that in the
past to micro-benchmark Mahout's legacy Math Linear Algebra backend.
Suneel
On Sun, Jul 17, 2016 at 11:57 AM, Ellison Anne Williams <
eawilliamspirk@gmail.com> wrote:
> Suneel -- Thanks for creating the JIRA issue and pointing out the licensing
> problems. I see that JMH is under the GNU GPL2 (
> http://openjdk.java.net/legal/) which is not compatible with the Apache
> license (http://www.apache.org/legal/resolved.html).
>
> It appears that Flink just removed the benchmarking code instead of
> re-porting it to another option.
>
> I would like us to port it to another license-compatible benchmarking
> framework such as Google Caliper (or something similar) instead of removing
> the code as the benchmarking is important for encryption optimization.
>
> Thoughts?
>
> Thanks!
>
> Ellison Anne
>