You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2022/04/28 07:16:49 UTC

[GitHub] [flink] Myasuka opened a new pull request, #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Myasuka opened a new pull request, #19600:
URL: https://github.com/apache/flink/pull/19600

   ## What is the purpose of the change
   
   Move RocksDB's log location to /tmp for e2e tests to avoid too many logs in e2e artifacts.
   
   
   ## Brief change log
   
   Configure `state.backend.rocksdb.log.dir` when starting the cluster.
   
   
   ## Verifying this change
   
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): no
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no
     - The serializers: no
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? no
     - If yes, how is the feature documented? not applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r860936180


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   isn't that like, really bad? Surely users would want some easy way to opt-out of rocks db dumping mountains of files into the log directory :/
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] Myasuka commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
Myasuka commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r865511866


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   If relocating to RocksDB's own folder, the folder itself will be cleaned up when the RocksDB state-backend disposed just as before. In other words, we don't need to manually clean up the log files but the parent folder will be cleaned up automatically.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol merged pull request #19600: [FLINK-27433][tests] Relocate RocksDB's log back to its own database dir

Posted by GitBox <gi...@apache.org>.
zentol merged PR #19600:
URL: https://github.com/apache/flink/pull/19600


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r861769777


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   I'd prefer that over writing to a tmp directory that we don't clean up.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] Myasuka commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
Myasuka commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r860922881


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   Even we set the log level as `HEADER_LEVEL`, it will still create a log file with configuration profiles.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] Myasuka commented on pull request #19600: [FLINK-27433][tests] Relocate RocksDB's log back to its own database dir

Posted by GitBox <gi...@apache.org>.
Myasuka commented on PR #19600:
URL: https://github.com/apache/flink/pull/19600#issuecomment-1120113084

   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] flinkbot commented on pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
flinkbot commented on PR #19600:
URL: https://github.com/apache/flink/pull/19600#issuecomment-1111838488

   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "f530b65d1b37920627ef96badfecd8aa719fe9da",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "f530b65d1b37920627ef96badfecd8aa719fe9da",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * f530b65d1b37920627ef96badfecd8aa719fe9da UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] Myasuka commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
Myasuka commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r860965808


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   If you want the logs still created under the DB folder, we can set this options to a non-existing folder, such as `/dev/null`. This will make the behavior same as before. Is this what you want?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [flink] zentol commented on a diff in pull request #19600: [FLINK-27433][tests] Move RocksDB's log location to /tmp for e2e tests

Posted by GitBox <gi...@apache.org>.
zentol commented on code in PR #19600:
URL: https://github.com/apache/flink/pull/19600#discussion_r860843361


##########
flink-end-to-end-tests/test-scripts/common.sh:
##########
@@ -305,6 +305,10 @@ function wait_dispatcher_running {
 }
 
 function start_cluster {
+  # After FLINK-24785, RocksDB's log would be created under Flink's log directory by default,
+  # this would make e2e tests' artifacts containing too many log files.
+  # As RocksDB's log would not help much in e2e tests, move the location to the '/tmp' folder.
+  set_config_key "state.backend.rocksdb.log.dir" "/tmp"

Review Comment:
   is there no way to disable the logging altogether? 🤔 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org