You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/20 10:10:22 UTC

[GitHub] [flink] zentol opened a new pull request, #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

zentol opened a new pull request, #19528:
URL: https://github.com/apache/flink/pull/19528

   Move as much out of misc as possible, and balance the 2 connectors profiles.
   
   -> runtime
   * gelly
   * statebackends
   * dstl
   * queryable-state
   
   -> connectors -> connectors_1
   * remaining filesystems/formats
   * hbase
   
   -> kafka/gelly -> connectors_2
   * pubsub
   * pulsar
   * rabitmq
   * aws
   
   -> misc
   * architecture-tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r858726683


##########
tools/ci/stage.sh:
##########
@@ -48,11 +52,23 @@ flink-metrics,\
 flink-metrics/flink-metrics-core,\
 flink-external-resources,\
 flink-external-resources/flink-external-resource-gpu,\
+flink-libraries,\
 flink-libraries/flink-cep,\
 flink-libraries/flink-cep-scala,\
-flink-libraries/flink-state-processing-api"
+flink-libraries/flink-state-processing-api,\
+flink-libraries/flink-gelly,\
+flink-libraries/flink-gelly-scala,\
+flink-libraries/flink-gelly-examples,\
+flink-queryable-state,\
+flink-queryable-state/flink-queryable-state-runtime,\
+flink-queryable-state/flink-queryable-state-client-java,\
+flink-container,\
+flink-dstl,\
+flink-dstl/flink-dstl-dfs,\

Review Comment:
   that's because `!` is a special character for shells; it's unrelated to maven.
   
   `mvn -pl !flink-core spotless:apply` will also fail for you. You need to escape the `!`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r858719635


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   any suggestions? there isn't a particular pattern (apart from formats/reporters/filesystems being in connector_1), and I'm not keen on a naming scheme that includes specific modules (e.g., "kafka/pulsar") as I'm not looking forward to renaming stuff when we externalize more stuff.
   
   Ultimately the lifetime of these stages is limited anyway; until the connectors have been externalized,



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r859489389


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   would connect_1 be short enough?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19528:
URL: https://github.com/apache/flink/pull/19528#issuecomment-1103766001

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "88f7a0ac18c893e13bbe0e61935accb47d81b02c",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "88f7a0ac18c893e13bbe0e61935accb47d81b02c",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 88f7a0ac18c893e13bbe0e61935accb47d81b02c UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r859454849


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   what about something like `stage_[1|2]_connectors` or `set_[1|2]_connectors` or just `[1|2]_connectors`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r859456323


##########
tools/ci/stage.sh:
##########
@@ -48,11 +52,23 @@ flink-metrics,\
 flink-metrics/flink-metrics-core,\
 flink-external-resources,\
 flink-external-resources/flink-external-resource-gpu,\
+flink-libraries,\
 flink-libraries/flink-cep,\
 flink-libraries/flink-cep-scala,\
-flink-libraries/flink-state-processing-api"
+flink-libraries/flink-state-processing-api,\
+flink-libraries/flink-gelly,\
+flink-libraries/flink-gelly-scala,\
+flink-libraries/flink-gelly-examples,\
+flink-queryable-state,\
+flink-queryable-state/flink-queryable-state-runtime,\
+flink-queryable-state/flink-queryable-state-client-java,\
+flink-container,\
+flink-dstl,\
+flink-dstl/flink-dstl-dfs,\

Review Comment:
   I see, I should have noticed the hint with `zsh: event not found: ,!`. But thanks for clarification. I missed the `\` in the negation. I interpreted it as a backspace for regex stuff... The backspace doesn't appear in the AzureCI output because `echo` will omit the escaping character...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on PR #19528:
URL: https://github.com/apache/flink/pull/19528#issuecomment-1110663491

   I didn't change anything in the force push; I just rebased the branch.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol merged pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol merged PR #19528:
URL: https://github.com/apache/flink/pull/19528


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r858719635


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   any suggestions? there isn't a particular pattern (apart from formats/reporters/filesystems being in connector_1), and I'm not keen on a naming scheme that includes specific modules (e.g., "kafka/pulsar") as I'm not keen on having to rename stuff when we externalize more stuff.
   
   Ultimately the lifetime of these stages is limited anyway; until the connectors have been externalized,



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on PR #19528:
URL: https://github.com/apache/flink/pull/19528#issuecomment-1109772621

   I did another analysis based on the following command revealing the modules that would still endup in misc:
   ```
   $ comm -3 <(cat tools/ci/stage.sh| head -172 | tail -n+31 | grep ^flink | sed "s/,\\\\$//" | sort | uniq | sort) <(find . -name "flink-*" -and -type d | grep "^./flink" | sed 's~./~~' | sort )
   	flink-architecture-tests
   	flink-architecture-tests/flink-architecture-tests-base
   	flink-architecture-tests/flink-architecture-tests-production
   	flink-architecture-tests/flink-architecture-tests-test
   	flink-connectors/flink-connector-elasticsearch5
   	flink-connectors/flink-connector-hive
   	flink-connectors/flink-connector-nifi
   	flink-connectors/flink-connector-twitter
   	flink-connectors/flink-sql-connector-hive-1.2.2
   	flink-connectors/flink-sql-connector-hive-2.2.0
   	flink-connectors/flink-sql-connector-hive-2.3.6
   	flink-connectors/flink-sql-connector-hive-2.3.9
   	flink-connectors/flink-sql-connector-hive-3.1.2
   	flink-contrib
   	flink-dist
   	flink-dist-scala
   	flink-dist/src/main/flink-bin
   	flink-dist/target/flink-1.16-SNAPSHOT-bin
   	flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
   	flink-docs
   	flink-end-to-end-tests
   	flink-end-to-end-tests/flink-batch-sql-test
   	flink-end-to-end-tests/flink-cli-test
   	flink-end-to-end-tests/flink-confluent-schema-registry
   	flink-end-to-end-tests/flink-connector-gcp-pubsub-emulator-tests
   	flink-end-to-end-tests/flink-dataset-allround-test
   	flink-end-to-end-tests/flink-dataset-fine-grained-recovery-test
   	flink-end-to-end-tests/flink-datastream-allround-test
   	flink-end-to-end-tests/flink-distributed-cache-via-blob-test
   	flink-end-to-end-tests/flink-elasticsearch5-test
   	flink-end-to-end-tests/flink-elasticsearch6-test
   	flink-end-to-end-tests/flink-elasticsearch7-test
   	flink-end-to-end-tests/flink-end-to-end-tests-aws-kinesis-firehose
   	flink-end-to-end-tests/flink-end-to-end-tests-aws-kinesis-streams
   	flink-end-to-end-tests/flink-end-to-end-tests-common
   	flink-end-to-end-tests/flink-end-to-end-tests-common-kafka
   	flink-end-to-end-tests/flink-end-to-end-tests-hbase
   	flink-end-to-end-tests/flink-end-to-end-tests-pulsar
   	flink-end-to-end-tests/flink-end-to-end-tests-scala
   	flink-end-to-end-tests/flink-file-sink-test
   	flink-end-to-end-tests/flink-glue-schema-registry-avro-test
   	flink-end-to-end-tests/flink-glue-schema-registry-json-test
   	flink-end-to-end-tests/flink-glue-schema-registry-test
   	flink-end-to-end-tests/flink-heavy-deployment-stress-test
   	flink-end-to-end-tests/flink-high-parallelism-iterations-test
   	flink-end-to-end-tests/flink-local-recovery-and-allocation-test
   	flink-end-to-end-tests/flink-metrics-availability-test
   	flink-end-to-end-tests/flink-metrics-reporter-prometheus-test
   	flink-end-to-end-tests/flink-netty-shuffle-memory-control-test
   	flink-end-to-end-tests/flink-parent-child-classloading-test-lib-package
   	flink-end-to-end-tests/flink-parent-child-classloading-test-program
   	flink-end-to-end-tests/flink-plugins-test
   	flink-end-to-end-tests/flink-python-test
   	flink-end-to-end-tests/flink-queryable-state-test
   	flink-end-to-end-tests/flink-quickstart-test
   	flink-end-to-end-tests/flink-rocksdb-state-memory-control-test
   	flink-end-to-end-tests/flink-sql-client-test
   	flink-end-to-end-tests/flink-state-evolution-test
   	flink-end-to-end-tests/flink-stream-sql-test
   	flink-end-to-end-tests/flink-stream-state-ttl-test
   	flink-end-to-end-tests/flink-stream-stateful-job-upgrade-test
   	flink-end-to-end-tests/flink-streaming-kafka-test
   	flink-end-to-end-tests/flink-streaming-kafka-test-base
   	flink-end-to-end-tests/flink-streaming-kinesis-test
   	flink-end-to-end-tests/flink-tpcds-test
   	flink-end-to-end-tests/flink-tpch-test
   	flink-examples
   	flink-examples/flink-examples-batch
   	flink-examples/flink-examples-build-helper
   	flink-examples/flink-examples-build-helper/flink-examples-streaming-gcp-pubsub
   	flink-examples/flink-examples-build-helper/flink-examples-streaming-state-machine
   	flink-examples/flink-examples-build-helper/flink-examples-streaming-twitter
   	flink-examples/flink-examples-streaming
   	flink-examples/flink-examples-table
   	flink-filesystems/flink-mapr-fs
   	flink-kubernetes
   	flink-python
   	flink-quickstart
   	flink-quickstart/flink-quickstart-java
   	flink-quickstart/flink-quickstart-scala
   	flink-scala-shell
   	flink-table/flink-table-uber
   	flink-test-utils-parent
   	flink-test-utils-parent/flink-connector-test-utils
   	flink-test-utils-parent/flink-connector-testing
   	flink-test-utils-parent/flink-test-utils-junit
   	flink-walkthroughs
   	flink-walkthroughs/flink-walkthrough-common
   	flink-walkthroughs/flink-walkthrough-datastream-java
   	flink-walkthroughs/flink-walkthrough-datastream-scala
   	flink-yarn
   	flink-yarn-tests
   ```
   there are still a few connector modules. Are these left out intentional?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r858687044


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   I don't like how these two stages have a common prefix. In AzureCI, the end of the name is sometimes cut of leaving both stages with the same label + "..." as a prefix. Can't we come up with something more descriptive?



##########
tools/ci/stage.sh:
##########
@@ -48,11 +52,23 @@ flink-metrics,\
 flink-metrics/flink-metrics-core,\
 flink-external-resources,\
 flink-external-resources/flink-external-resource-gpu,\
+flink-libraries,\
 flink-libraries/flink-cep,\
 flink-libraries/flink-cep-scala,\
-flink-libraries/flink-state-processing-api"
+flink-libraries/flink-state-processing-api,\
+flink-libraries/flink-gelly,\
+flink-libraries/flink-gelly-scala,\
+flink-libraries/flink-gelly-examples,\
+flink-queryable-state,\
+flink-queryable-state/flink-queryable-state-runtime,\
+flink-queryable-state/flink-queryable-state-client-java,\
+flink-container,\
+flink-dstl,\
+flink-dstl/flink-dstl-dfs,\

Review Comment:
   The succeeding commas seem to be fine for `mvn -pl` on the modules. But negating a list like `flink-core,` will result in something like `mvn -pl !flink-core,!` will cause the maven call to fail...
   
   It fails on my local machine (MacOS) at least:
   ```
   $ mvn -pl flink-core,!,!flink-runtime spotless:apply
   zsh: event not found: ,!
   ```
   
   Interestingly, the [AzureCI run](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34881&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23160) succeeds with a module definition like `[...]!flink-dstl,!flink-dstl/flink-dstl-dfs,!,!flink-table[...]` being in place. Maybe, a OS issue?
   
   We might want to fix it anyway...



##########
tools/ci/stage.sh:
##########
@@ -48,11 +52,23 @@ flink-metrics,\
 flink-metrics/flink-metrics-core,\
 flink-external-resources,\
 flink-external-resources/flink-external-resource-gpu,\
+flink-libraries,\
 flink-libraries/flink-cep,\
 flink-libraries/flink-cep-scala,\
-flink-libraries/flink-state-processing-api"
+flink-libraries/flink-state-processing-api,\
+flink-libraries/flink-gelly,\
+flink-libraries/flink-gelly-scala,\
+flink-libraries/flink-gelly-examples,\
+flink-queryable-state,\
+flink-queryable-state/flink-queryable-state-runtime,\
+flink-queryable-state/flink-queryable-state-client-java,\
+flink-container,\
+flink-dstl,\
+flink-dstl/flink-dstl-dfs,\

Review Comment:
   FYI: This is not the only location with this error. I was just too lazy to mark all of them 😇 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
zentol commented on PR #19528:
URL: https://github.com/apache/flink/pull/19528#issuecomment-1109800909

   > there are still a few connector modules [in misc]. Are these left out intentional?
   
   nifi, twitter and es5 have already been removed from master; you just have leftover directories. Hive is intentionally left in misc because it is too expensive to move to the connector profiles without tipping the balance too much.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on PR #19528:
URL: https://github.com/apache/flink/pull/19528#issuecomment-1110622103

   I see, thanks for clarification. nifi, twitter, and es5 are, indeed, leftover modules. I should have checked it first... 👍 The other modules look reasonable for the misc module in my opinion...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] XComp commented on a diff in pull request #19528: [FLINK-27228][ci] Redistributed modules across CI profiles

Posted by GitBox <gi...@apache.org>.
XComp commented on code in PR #19528:
URL: https://github.com/apache/flink/pull/19528#discussion_r859581561


##########
tools/ci/stage.sh:
##########
@@ -21,8 +21,8 @@ STAGE_COMPILE="compile"
 STAGE_CORE="core"
 STAGE_PYTHON="python"
 STAGE_TABLE="table"
-STAGE_CONNECTORS="connectors"
-STAGE_KAFKA_GELLY="kafka/gelly"
+STAGE_CONNECTORS_1="connectors_1"
+STAGE_CONNECTORS_2="connectors_2"

Review Comment:
   I mean, in the end it depends on the screen and font size of the user. One could also argue that `test_ci` is already problematic as a prefix for the setup I had in mind. Considering that (and agreeing that the index being at the end of the name is more naturally) I'd be ok with just keeping it like that for now. Probably, it's just overengineering. If it's really an issue we could address it later on...



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org