You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Michael Luckey (JIRA)" <ji...@apache.org> on 2019/02/11 00:07:00 UTC

[jira] [Comment Edited] (BEAM-4046) Decouple gradle project names and maven artifact ids

    [ https://issues.apache.org/jira/browse/BEAM-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16764604#comment-16764604 ] 

Michael Luckey edited comment on BEAM-4046 at 2/11/19 12:06 AM:
----------------------------------------------------------------

Hi [~kenn],

looking (shortly) into that, we should shortly discuss, whether we still want to do that, and if I understood right. As it will imply tons of string replacements to align with changed project names which just take some time to get it right, I did not want to start a PR without reassuring this is still valid.

IIUC - and I might realise, I am totally wrong here during implementation (*) - this could look something as `settings.gradle`
{noformat}
rootProject.name = "beam"

include ":release"

include ":examples:java"
include ":model:fn-execution"
include ":model:job-management"
include ":model:pipeline"
include ":runners:apex"
include ":runners:core-construction-java"
include ":runners:core-java"
include ":runners:direct-java"
include ":runners:extensions-java:metrics"
include ":runners:flink" // FIXME was flink_2.11
include ":runners-flink:job-server" // FIXME was flink_2.11
include ":runners-flink:job-server-container" // FIXME was flink_2.11
include ":runners:flink:1.6"
include ":runners:flink:1.6:job-server"
include ":runners:flink:1.6:job-server-container"
include ":runners:flink:1.7"
include ":runners:flink:1.7:job-server"
include ":runners:flink:1.7:job-server-container"
include ":runners:gearpump"
include ":runners:google-cloud-dataflow-java"
include ":runners-google-cloud-dataflow-java:examples"
include ":runners:google-cloud-dataflow-java:examples-streaming"
include ":runners:java-fn-execution"
include ":runners:local-java" // FIXME
include ":runners:reference:java"
include ":runners:reference:job-server"
include ":runners:spark"
include ":runners:samza"
include ":sdks:go"
include ":sdks:go:container"
include ":sdks:go:examples"
include ":sdks:go:test"
include ":sdks:java:build-tools"
include ":sdks:java:container"
include ":sdks:java:core"
include ":sdks:java:extensions:euphoria"
include ":sdks:java:extensions:kryo"
include ":sdks:java:extensions:google-cloud-platform-core"
include ":sdks:java:extensions:jackson" // FIXME
include ":sdks:java:extensions:join-library"
include ":sdks:java:extensions:protobuf"
include ":sdks:java:extensions:sketching"
include ":sdks:java:extensions:sorter"
include ":sdks:java:extensions:sql"
include ":sdks:java:extensions:sql:jdbc"
include ":sdks:java:extensions:sql:shell"
include ":sdks:java:fn-execution"
include ":sdks:java:harness"
include ":sdks:java:io:amazon-web-services"
include ":sdks:java:io:amqp"
include ":sdks:java:io:cassandra"
include ":sdks:java:io:clickhouse"
include ":sdks:java:io:common"
include ":sdks:java:io:elasticsearch"
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-2" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-5" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-6" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common" // FIXME
include ":sdks:java:io:file-based-io-tests"
include ":sdks:java:io:google-cloud-platform"
include ":sdks:java:io:hadoop-common"
include ":sdks:java:io:hadoop-file-system"
include ":sdks:java:io:hadoop-input-format"
include ":sdks:java:io:hadoop-format"
include ":sdks:java:io:hbase"
include ":sdks:java:io:hcatalog"
include ":sdks:java:io:jdbc"
include ":sdks:java:io:jms"
include ":sdks:java:io:kafka"
include ":sdks:java:io:kinesis"
include ":sdks:java:io:kudu"
include ":sdks:java:io:mongodb"
include ":sdks:java:io:mqtt"
include ":sdks:java:io:parquet"
include ":sdks:java:io:rabbitmq"
include ":sdks:java:io:redis"
include ":sdks:java:io:solr"
include ":sdks:java:io:tika"
include ":sdks:java:io:xml"
include ":sdks:java:io:synthetic"
include ":sdks:java:javadoc"
include ":sdks:java:testing:load-tests"
include ":sdks:java:maven-archetypes:examples"
include ":sdks:java:maven-archetypes:starter"
include ":sdks:java:testing:nexmark"
include ":sdks:python"
include ":sdks:python:container"
include ":sdks:python:container:py3"
include ":sdks:python:precommit:dataflow"
include ":sdks:python-precommit:dataflow:py3"
include ":sdks:python:precommit:direct:py3"
include ":vendor:grpc-1_13_1"
include ":sdks:java:testing:test-utils"
include ":vendor:sdks-java-extensions-protobuf"
include ":vendor:guava-20_0"
include ":website"
include ":runners:google-cloud-dataflow-java:worker:legacy-worker" // FIXME
include ":runners:google-cloud-dataflow-java:worker" // FIXME
include ":runners:google-cloud-dataflow-java:worker:windmill" // FIXME
include ":.test-infra:metrics"
include ":sdks:java:bom"
{noformat}
and small addition in beam-module plugin
{code}
    project.apply plugin: 'base'
    project.archivesBaseName = 'beam' + project.path.replace(':', '-')
{code}
`archivesBaseName` defaults to `project.name` which probably is exactly what we stumbled upon.
 Mind the fixmes. These are special cases, where the folder structure does not match 
 the project name. Here we will need to overwrite explicitly in the modules build.gradle like e.g.
{code}
    apply plugin: 'base'
    archivesBaseName = 'beam-runners-flink_2.11-job-server-container' 
{code}
I have not yet tested if it will work throughout the project. We will need to change all that project references, as
 they have changed. And we have some places where we rely on `project.name` which most likely have to be adopted.

Of course, that will also change the way, tasks are defined (and executed on cli)... so we might break some devs
 shortcuts/aliases/scripts... dunno. Which in turn might imply, this is an api breaking change, we should not do?
 What do you think about it?

(*) For instance we might have task/external plugins which use project name in a way, that breaks this. Or some
 custom archive task, which does not comply to that api... you never know for sure


was (Author: michel):
Hi [~kenn],

looking (shortly) into that, we should shortly discuss, whether we still want to do that, and if I understood right.
As it will imply tons of string replacements to align with changed project names which jsut take some time
to get it right.

IIUC - and I might realise, I am totally wrong here during implementation (*) - this could look something as
`settings.gradle`

{noformat}
rootProject.name = "beam"

include ":release"

include ":examples:java"
include ":model:fn-execution"
include ":model:job-management"
include ":model:pipeline"
include ":runners:apex"
include ":runners:core-construction-java"
include ":runners:core-java"
include ":runners:direct-java"
include ":runners:extensions-java:metrics"
include ":runners:flink" // FIXME was flink_2.11
include ":runners-flink:job-server" // FIXME was flink_2.11
include ":runners-flink:job-server-container" // FIXME was flink_2.11
include ":runners:flink:1.6"
include ":runners:flink:1.6:job-server"
include ":runners:flink:1.6:job-server-container"
include ":runners:flink:1.7"
include ":runners:flink:1.7:job-server"
include ":runners:flink:1.7:job-server-container"
include ":runners:gearpump"
include ":runners:google-cloud-dataflow-java"
include ":runners-google-cloud-dataflow-java:examples"
include ":runners:google-cloud-dataflow-java:examples-streaming"
include ":runners:java-fn-execution"
include ":runners:local-java" // FIXME
include ":runners:reference:java"
include ":runners:reference:job-server"
include ":runners:spark"
include ":runners:samza"
include ":sdks:go"
include ":sdks:go:container"
include ":sdks:go:examples"
include ":sdks:go:test"
include ":sdks:java:build-tools"
include ":sdks:java:container"
include ":sdks:java:core"
include ":sdks:java:extensions:euphoria"
include ":sdks:java:extensions:kryo"
include ":sdks:java:extensions:google-cloud-platform-core"
include ":sdks:java:extensions:jackson" // FIXME
include ":sdks:java:extensions:join-library"
include ":sdks:java:extensions:protobuf"
include ":sdks:java:extensions:sketching"
include ":sdks:java:extensions:sorter"
include ":sdks:java:extensions:sql"
include ":sdks:java:extensions:sql:jdbc"
include ":sdks:java:extensions:sql:shell"
include ":sdks:java:fn-execution"
include ":sdks:java:harness"
include ":sdks:java:io:amazon-web-services"
include ":sdks:java:io:amqp"
include ":sdks:java:io:cassandra"
include ":sdks:java:io:clickhouse"
include ":sdks:java:io:common"
include ":sdks:java:io:elasticsearch"
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-2" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-5" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-6" // FIXME
include ":sdks:java:io:elasticsearch-tests:elasticsearch-tests-common" // FIXME
include ":sdks:java:io:file-based-io-tests"
include ":sdks:java:io:google-cloud-platform"
include ":sdks:java:io:hadoop-common"
include ":sdks:java:io:hadoop-file-system"
include ":sdks:java:io:hadoop-input-format"
include ":sdks:java:io:hadoop-format"
include ":sdks:java:io:hbase"
include ":sdks:java:io:hcatalog"
include ":sdks:java:io:jdbc"
include ":sdks:java:io:jms"
include ":sdks:java:io:kafka"
include ":sdks:java:io:kinesis"
include ":sdks:java:io:kudu"
include ":sdks:java:io:mongodb"
include ":sdks:java:io:mqtt"
include ":sdks:java:io:parquet"
include ":sdks:java:io:rabbitmq"
include ":sdks:java:io:redis"
include ":sdks:java:io:solr"
include ":sdks:java:io:tika"
include ":sdks:java:io:xml"
include ":sdks:java:io:synthetic"
include ":sdks:java:javadoc"
include ":sdks:java:testing:load-tests"
include ":sdks:java:maven-archetypes:examples"
include ":sdks:java:maven-archetypes:starter"
include ":sdks:java:testing:nexmark"
include ":sdks:python"
include ":sdks:python:container"
include ":sdks:python:container:py3"
include ":sdks:python:precommit:dataflow"
include ":sdks:python-precommit:dataflow:py3"
include ":sdks:python:precommit:direct:py3"
include ":vendor:grpc-1_13_1"
include ":sdks:java:testing:test-utils"
include ":vendor:sdks-java-extensions-protobuf"
include ":vendor:guava-20_0"
include ":website"
include ":runners:google-cloud-dataflow-java:worker:legacy-worker" // FIXME
include ":runners:google-cloud-dataflow-java:worker" // FIXME
include ":runners:google-cloud-dataflow-java:worker:windmill" // FIXME
include ":.test-infra:metrics"
include ":sdks:java:bom"
{noformat}

and small addition in beam-module plugin

{code:groovy}
    project.apply plugin: 'base'
    project.archivesBaseName = 'beam' + project.path.replace(':', '-')
{code}

`archivesBaseName` defaults to `project.name` which probably is exactly what we stumbled upon.
Mind the fixmes. These are special cases, where the folder structure does not match 
the project name. Here we will need to overwrite explicitly in the modules build.gradle like e.g.

{code:groovy}
    apply plugin: 'base'
    archivesBaseName = 'beam-runners-flink_2.11-job-server-container' 
{code}

I have not yet tested if it will work throughout the project. We will need to change all that project references, as
they have changed. And we have some places where we rely on `project.name` which most likely have to be adopted.

Of course, that will also change the way, tasks are defined (and executed on cli)... so we might break some devs
shortcuts/aliases/scripts... dunno. Which in turn might imply, this is an api breaking change, we should not do?
What do you think about it?

(*) For instance we might have task/external plugins which use project name in a way, that breaks this. Or some
custom archive task, which does not comply to that api... you never know for sure


> Decouple gradle project names and maven artifact ids
> ----------------------------------------------------
>
>                 Key: BEAM-4046
>                 URL: https://issues.apache.org/jira/browse/BEAM-4046
>             Project: Beam
>          Issue Type: Sub-task
>          Components: build-system
>            Reporter: Kenneth Knowles
>            Priority: Major
>
> In our first draft, we had gradle projects like {{":beam-sdks-java-core"}}. It is clumsy and requires a hacky settings.gradle that is not idiomatic.
> In our second draft, we changed them to names that work well with Gradle, like {{":sdks:java:core"}}. This caused Maven artifact IDs to be wonky.
> In our third draft, we regressed to the first draft to get the Maven artifact ids right.
> These should be able to be decoupled. It seems there are many StackOverflow questions on the subject.
> Since it is unidiomatic and a poor user experience, if it does turn out to be mandatory then it needs to be documented inline everywhere - the settings.gradle should say why it is so bizarre, and each build.gradle should indicate what its project id is.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)