You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/03/25 03:09:37 UTC

[GitHub] [druid] suneet-s opened a new pull request #9560: Instructions to run integration tests against quickstart

suneet-s opened a new pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560
 
 
   This change makes it easier to run integration tests against a quickstart cluster running on a local development machine.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604118271
 
 
   > > @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test".
   > > I feel like that "other integration test" instead of doing exclude should just do include
   > 
   > The rationale for excluding was that it's better to accidentally run tests in multiple jobs rather than having a test run in no jobs, but we can adjust the approach if needed.
   
   That's true but I think as we add more and more IT, especially those unique ones that needs special setup, that exclude list will be super long and ugly  

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh merged pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh merged pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560
 
 
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-603635723
 
 
   What is the motivation for running tests Using A Quickstart Cluster?  Is it just mainly for debuggability? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604115060
 
 
   @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test"

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604116684
 
 
   > @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test".
   > I feel like that "other integration test" instead of doing exclude should just do include
   
   The rationale for excluding was that it's better to accidentally run tests in multiple jobs rather than having a test run in no jobs, but we can adjust the approach if needed.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398186471
 
 

 ##########
 File path: integration-tests/README.md
 ##########
 @@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
 can either be 8 or 11.
 
 Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>. 
-The file must contain one property per line, the key must start with druid_ and the format should be snake case. 
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case. 
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398202520
 
 

 ##########
 File path: integration-tests/src/test/java/org/apache/druid/tests/TestNGGroup.java
 ##########
 @@ -33,29 +33,57 @@
   // This group can only be run individually using -Dgroups=query since it requires specific test data setup.
   public static final String QUERY = "query";
   public static final String REALTIME_INDEX = "realtime-index";
-  // This group can only be run individually using -Dgroups=security since it requires specific test data setup.
+
+  /**
+   * This group can only be run individually using -Dgroups=security since it requires specific test data setup.
+   */
   public static final String SECURITY = "security";
-  // This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
-  // The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
-  // See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
+
+  /**
+   * This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
+   * The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
+   * See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
+   */
   public static final String S3_DEEP_STORAGE = "s3-deep-storage";
-  // This group is not part of CI. To run this group, gcs configs/credentials for your gcs must be provided in a file.
-  // The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
-  // See integration-tests/docker/environment-configs/override-examples/gcs for env vars to provide.
-  // The path to the folder that contains your GOOGLE_APPLICATION_CREDENTIALS file must also be pass
-  // to mvn with -Dresource.file.dir.path=<PATH_TO_FOLDER>
+
+  /**
+   * This group is not part of CI. To run this group, gcs configs/credentials for your gcs must be provided in a file.
+   * The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
+   * See integration-tests/docker/environment-configs/override-examples/gcs for env vars to provide.
+   * The path to the folder that contains your GOOGLE_APPLICATION_CREDENTIALS file must also be pass
+   * to mvn with -Dresource.file.dir.path=<PATH_TO_FOLDER>
+   */
   public static final String GCS_DEEP_STORAGE = "gcs-deep-storage";
-  // This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
-  // The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
-  // See integration-tests/docker/environment-configs/override-examples/azures for env vars to provide.
+
+  /**
+   * This group is not part of CI. To run this group, azure configs/credentials for your azure must be provided in a file.
+   * The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
+   * See integration-tests/docker/environment-configs/override-examples/azures for env vars to provide.
+   */
   public static final String AZURE_DEEP_STORAGE = "azure-deep-storage";
-  // This group is not part of CI. To run this group, hadoop configs must be provided in a file. The path of the file
-  // must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
-  // See integration-tests/docker/environment-configs/override-examples/hdfs for env vars to provide.
-  // Additionally, hadoop docker must be started by passing -Dstart.hadoop.docker=true to mvn.
+
+  /**
+   * This group is not part of CI. To run this group, hadoop configs must be provided in a file. The path of the file
+   * must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
+   * See integration-tests/docker/environment-configs/override-examples/hdfs for env vars to provide.
+   * Additionally, hadoop docker must be started by passing -Dstart.hadoop.docker=true to mvn.
+   */
   public static final String HDFS_DEEP_STORAGE = "hdfs-deep-storage";
-  // This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
-  // The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
-  // See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
+
+  /**
+   * This group is not part of CI. To run this group, s3 configs/credentials for your s3 must be provided in a file.
+   * The path of the file must then be pass to mvn with -Doverride.config.path=<PATH_TO_FILE>
+   * See integration-tests/docker/environment-configs/override-examples/s3 for env vars to provide.
+   */
   public static final String S3_INGESTION = "s3-ingestion";
+
+  /**
+   * This group os not part of CI explicitly. It allows you to run all the tests that have been tested with
 
 Review comment:
   If you push another commit, there's a small typo here: group os -> group is

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604225864
 
 
   I don't see it in the list of tests run by the "other" test job. I think it's excluded because of this in the config `-DexcludedGroups=batch-index`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604119377
 
 
   > @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test".
   > I feel like that "other integration test" instead of doing exclude should just do include
   
   Actually, I think this might be wrong. The test is assigned to 2 groups. So if I exclude 1/2 groups, I think it might exclude the test. Need to check locally if that's the case. So much TestNG magic I'm learning about in this PR 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 edited a comment on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604115060
 
 
   @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test".
   I feel like that "other integration test" instead of doing exclude should just do include

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398089064

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>.
-The file must contain one property per line, the key must start with druid_ and the format should be snake case.
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case.
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
+machine.
+
+Note that not all features run by default on a quickstart cluster, so it may

Review comment:
I feel the term "feature" might be a little misleading. Some of the setup on the docker cluster are just pre-inserted data into the metadata store or certain files place at certain locations. The feature itself like pulling segments from pathSpec in metadatastore is always there. I think as we add more and more IT and/or setup scripts, it will also be hard to knows which can work out of the box on quickstart vs. which can only work on docker cluster. I think it might help if we can list out which groups can work out of the box on quickstart vs. which can only work on docker cluster in the README. In the file with the group definition (TestNGGroup) we can then list out special setup needed (if needed) for those groups.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398196968

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Review comment:
👍 Added a note on the setup, but it's kinda hard to know which groups depend on what right now, maybe after using this for a few months the test group will be more complete. I added instructions to this file and the group so that other devs can add more tests to this list.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-603903816
 
 
   @maytasm3 Yes - debuggability and speed.
   
   I was trying to debug the integration tests. Unfortunately the docker environment is spun up and down as part of the integration tests, so it's hard to debug that environment. It's also quicker for me to test against since I usually have quickstart of a few different versions of druid on my dev machine. This makes it easier for me to verify that the integration test I'm adding fails for a new version, but works with my fix.
   
   Also spinning up the docker environment can take some time, so I found it faster to point my tests to an already running quickstart

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398072146

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Review comment:
Suggestion: Add `:` at the end

(similar comment for other occurrences below)

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398082242

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Review comment:
@maytasm3 I was worried about what you mentioned as a top level comment on this PR too. I tried to talk about it here, but maybe the warning is not big and scary enough. Any suggestions on how to make it more prominent? Also I wonder if I should point to the docker setup scripts for the next person who might find a test that fails when run against quickstart.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398188200

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Review comment:
Done

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398073266

##########
File path: integration-tests/README.md
##########
@@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
can either be 8 or 11.

Review comment:
Having to create a new directory as a child of root seems strange. Is there a way to make it local to the druid repo directory?

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398188256
 
 

 ##########
 File path: integration-tests/README.md
 ##########
 @@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
 can either be 8 or 11.
 
 Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>. 
-The file must contain one property per line, the key must start with druid_ and the format should be snake case. 
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case. 
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
+machine.
+
+Note that not all features run by default on a quickstart cluster, so it may
+not make sense to run the entire test suite against this configuration.
+
+Make sure you have at least 6GB of memory available before you run the tests.
+
+The tests rely on files in the test/resources folder to exist under the path /resources,
+so create a symlink to make them available
+
+```
+  ln -s ${DRUID_HOME}/integration-tests/src/test/resources /resources
+```
+
+Set the cluster config file environment variable to the quickstart config
+```
+  export CONFIG_FILE=${DRUID_HOME}/integration-tests/quickstart-it.json
+```
+Note that quickstart does not run with ssl, so to trick the integration tests
+we specify the `*_tls_url` in the config to be the same as the http url
 
 Review comment:
   Done

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398085489
 
 

 ##########
 File path: integration-tests/README.md
 ##########
 @@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
 can either be 8 or 11.
 
 Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>. 
-The file must contain one property per line, the key must start with druid_ and the format should be snake case. 
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case. 
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
+machine.
+
+Note that not all features run by default on a quickstart cluster, so it may
 
 Review comment:
   Other places in the docs use a blockquote like: https://druid.apache.org/docs/latest/querying/lookups.html

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604167731
 
 
   > > @suneet-s You should also exclude the new group QUICKSTART_COMPATIBLE from travis "other integration test".
   > > I feel like that "other integration test" instead of doing exclude should just do include
   > 
   > Actually, I think this might be wrong. The test is assigned to 2 groups. So if I exclude 1/2 groups, I think it might exclude the test. Need to check locally if that's the case. So much TestNG magic I'm learning about in this PR
   
   @maytasm3 I tried this locally, and it appears to run the test.
   
   `mvn clean verify -P int-tests-config-file -Dit.test=ITCompactionTaskTest -DexcludedGroups=quickstart-compatible`
   
   I'll look at the results of this travis run to make sure that ITCompactionTaskTest is actually run in this change

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398073920
 
 

 ##########
 File path: integration-tests/README.md
 ##########
 @@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
 can either be 8 or 11.
 
 Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>. 
-The file must contain one property per line, the key must start with druid_ and the format should be snake case. 
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case. 
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
+machine.
+
+Note that not all features run by default on a quickstart cluster, so it may
+not make sense to run the entire test suite against this configuration.
+
+Make sure you have at least 6GB of memory available before you run the tests.
+
+The tests rely on files in the test/resources folder to exist under the path /resources,
+so create a symlink to make them available
+
+```
+  ln -s ${DRUID_HOME}/integration-tests/src/test/resources /resources
+```
+
+Set the cluster config file environment variable to the quickstart config
+```
+  export CONFIG_FILE=${DRUID_HOME}/integration-tests/quickstart-it.json
+```
+Note that quickstart does not run with ssl, so to trick the integration tests
+we specify the `*_tls_url` in the config to be the same as the http url
 
 Review comment:
   Suggestion: Add `.` at the end

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-603635723
 
 
   What is the motivation for running tests Using A Quickstart Cluster? 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

ccaominh commented on a change in pull request #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#discussion_r398071640
 
 

 ##########
 File path: integration-tests/README.md
 ##########
 @@ -66,7 +66,40 @@ Integration tests can also be run with either Java 8 or Java 11 by adding -Djvm.
 can either be 8 or 11.
 
 Druid's configuration (using Docker) can be overrided by providing -Doverride.config.path=<PATH_TO_FILE>. 
-The file must contain one property per line, the key must start with druid_ and the format should be snake case. 
+The file must contain one property per line, the key must start with `druid_` and the format should be snake case. 
+
+Running Tests Using A Quickstart Cluster
+-------------------
+
+When writing integration tests, it can be helpful to test against a quickstart
+cluster so that you can set up remote debugging with in your developer
+environment. This section walks you through setting up the integration tests
+so that it can run against a quickstart cluster running on your development
 
 Review comment:
   How about linking against the quickstart docs so that folks know what is meant by a "quickstart cluster"?

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604225009
 
 
   Does this means that ITIndexerTest will get run twice? (once in batch index and once more in other)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] suneet-s commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

suneet-s commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604221332
 
 
   The `ITCompactionTaskTest` was excluded in the other integration test run - https://travis-ci.org/github/apache/druid/jobs/667053767?utm_medium=notification&utm_source=github_status, so I reverted the change that explicitly excludes that test group and added more javadocs.
   
   The `ITIndexerTest` was run coorectly - https://travis-ci.org/github/apache/druid/jobs/667053761?utm_medium=notification&utm_source=github_status

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart

Posted by GitBox <gi...@apache.org>.

maytasm3 commented on issue #9560: Instructions to run integration tests against quickstart
URL: https://github.com/apache/druid/pull/9560#issuecomment-604010024
 
 
   @suneet-s I think this is super useful as you can easily have more control over Druid cluster such as setting up remote debugging. One problem I see is that the docker environment do some set up of its own that is needed for some integration tests. For example, some confs set and tls/crl files generated for security IT or inserting rows into druid_segments metadatastore for the query IT (and also aws s3 keys to pull those segments). These most likely will not be configured in quickstart cluster and can result in false failures. Maybe it's worth mentioning the potential problems in the README so user does not except full compatibility out of the box

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org