You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2022/05/04 14:17:37 UTC

[GitHub] [druid] FrankChen021 opened a new pull request, #12501: Move web-console dependency declaration from druid-server to druid-distribution

FrankChen021 opened a new pull request, #12501:
URL: https://github.com/apache/druid/pull/12501

   ### Description
   
   This is a follow up of #12486 
   
   #12486 removes the web-console dependency from druid-server module because the dependency check reports that it's not used in druid-server module.
   
   Because druid-server module is referenced by many other modules, the existence of web-console dependency in druid-server module makes that module to be compiled after the web-console module. However, building of web-console module usually takes a long time. If the whole module is built in parallel with `-T` mvn parameter, this dependency makes the building slower.
   
   After #12486 is merged, @vogievetsky found that this dependency must be there or the web-console module won't be shipped in the distribution package(https://github.com/apache/druid/pull/12486#discussion_r863492827)
   
   The druid distribution package is packaged by the `mvn-assembly-plugin` referenced in the `druid-distribution` module. 
   https://github.com/apache/druid/blob/0206a2da5c6211caf451751876d847f40c2e7755/distribution/pom.xml#L256-L258
   
   This plugin is also configured to copy all dependencies declared in the pom of `druid-distribution` to the `lib` directory as below.
   https://github.com/apache/druid/blob/0206a2da5c6211caf451751876d847f40c2e7755/distribution/src/assembly/assembly.xml#L126-L129
   
   So, we can move the dependency declaration from the druid-server module to the druid-distribution module so that we still gain the  building speed up as well as the package integrity.
   
   Also, to prevent the missing of web-console in future, I added an IT case to check if the default home page of the web-console exists.
   
   
   This PR has:
   - [X] been self-reviewed.
      - [ ] using the [concurrency checklist](https://github.com/apache/druid/blob/master/dev/code-review/concurrency.md) (Remove this item if the PR doesn't have any relation to concurrency.)
   - [ ] added documentation for new or modified features or behaviors.
   - [ ] added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
   - [ ] added or updated version, license, or notice information in [licenses.yaml](https://github.com/apache/druid/blob/master/dev/license.md)
   - [ ] added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
   - [ ] added unit tests or modified existing tests to cover new code paths, ensuring the threshold for [code coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md) is met.
   - [X] added integration tests.
   - [X] been tested in a test Druid cluster.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251758556

   > I'm wondering if we can use 'profile' to include the web-console in the server/pom.xml and active such profile from intellij when debugging?
   
   I think it's possible. Intellij provides a profile panel that allows us to active specific profiles. 
   
   <img width="444" alt="image" src="https://user-images.githubusercontent.com/6525742/191153125-a32562f5-e4b8-4673-b338-7e95bd30054b.png">
   
   We can keep that newly added profile selected from Intellij. @clintropolis Do you have any other ideas?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251753801

   I'm wondering if we can use 'profile' to include the web-console in the server/pom.xml and active such profile from intellij when debugging?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251749118

   Hello @clintropolis 
   
   Sorry to hear that. I didn't use debug from Intelli to lanch process, instead of I usually use 'attach' to an existing running process.
   
   Since web-console is served byt router, do you mean you debug router in this way?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1246199678

   Hello @clintropolis Any update on this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251758948

   I think I've got it working well enough using `-Xbootclasspath` to customize what the router has access to when it starts up. Yeah, I have a handful of xml files that I use to launch druid processes directly within intellij. Extensions (and now the web-console) are a bit funny in this model as you have to externally build them and refer to their location, but works pretty well otherwise.
   
   Here is my `Router.xml` for example (after fixing):
   ```
   <component name="ProjectRunConfigurationManager">
     <configuration default="false" name="Router" type="Application" factoryName="Application">
       <option name="ALTERNATIVE_JRE_PATH" value="1.8" />
       <option name="MAIN_CLASS_NAME" value="org.apache.druid.cli.Main" />
       <module name="druid-services" />
       <option name="PROGRAM_PARAMETERS" value="server router" />
       <option name="VM_PARAMETERS" value="-server -Xbootclasspath/a:$PROJECT_DIR$/conf/router/:$PROJECT_DIR$/web-console/target/web-console-25.0.0-SNAPSHOT.jar -Duser.timezone=UTC -Dfile.encoding=UTF-8 -Xmx256M -Xmx256M -XX:+UseG1GC -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dorg.jboss.logging.provider=slf4j -Dlog4j.configurationFile=$PROJECT_DIR$/core/src/main/resources/log4j2.debug.xml -Ddruid.host=localhost -Ddruid.zk.service.host=localhost -Ddruid.emitter=logging -Ddruid.router.managementProxy.enabled=true -Ddruid.extensions.directory=$PROJECT_DIR$/distribution/target/extensions/ -Ddruid.extensions.loadList=[\&quot;druid-histogram\&quot;,\&quot;druid-bloom-filter\&quot;,\&quot;druid-datasketches\&quot;,\&quot;druid-lookups-cached-global\&quot;,\&quot;druid-basic-security\&quot;] -Ddruid.server.http.numThreads=50  -Ddruid.generic.useDefaultValueForNull=false " />
     </configuration>
   </component>
   ```
   
   If you put these xml files in `.idea/runConfigurations/` then you have all of these options to run various services in the run/debug dropdown menu:
   <img width="848" alt="Screen Shot 2022-09-19 at 7 13 42 PM" src="https://user-images.githubusercontent.com/1577461/191152314-b0105df5-6cc9-4062-b98b-3ceb57f4d50f.png">
   
   I do this because in my experience its got a faster turn-around than building a quickstart and attaching, can modify most stuff on the fly and live reload, etc.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251762512

   Glad to hear it works. Sorry for the inconvenience caused by this change. @clintropolis 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1144340392

   @paul-rogers Thanks for the review. 
   
   I need to make some changes to the CI. Since the web-console is moved to `distribution`, building of IT module does not involve packaging the web-console module, the newly added test can not pass. To make the test in a separated group can let it run independently in a separated CI job, in that way, other jobs don't need to packaging the web-console module which can save time.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1242701658

   Since #12776 has moved web-console e2e test to phase 1, there's no need to an IT test anymore, revert these changes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251726819

   hmm, I think this might have broken how I was debugging druid services in intellij using run configurations (similar to https://github.com/apache/druid/blob/master/dev/intellij-setup.md#coordinatorxml), since the web-console is no longer working for me. I haven't verified that it is this change yet, but it seems pretty likely given that the above setup is not using the distribution packaging at all.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
clintropolis commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1251730597

   >I haven't verified that it is this change yet, but it seems pretty likely given that the above setup is not using the distribution packaging at all.
   
   confirmed it is related to this change, I'll see if I can find a solution to my debugger problem, since I use that method of debugging fairly frequently...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1247496138

   Thank you @clintropolis 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] paul-rogers commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
paul-rogers commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1144256459

   @FrankChen021, thanks for making this fix: the revised dependency looks right: it should be `distribution` that depends on the console, not some module which `distribution` depends on. That is, make the dependency direct, not transitive.
   
   Adding an IT is usually a Good Thing. There is, however, a big cost to adding a test group in the current setup: we build the entire Druid stack for each test group. I've been chipping away at [First cut at restructuring the integration tests](https://github.com/apache/druid/pull/12368) to make the ITs simpler and faster: your new test is a good candidate to port.
   
   In the mean while, can the IT be added to an existing group? It can just piggy-back on any one of the existing clusters, can't it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1242873795

   Hello @clintropolis @vogievetsky Could you review this PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 commented on pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on PR #12501:
URL: https://github.com/apache/druid/pull/12501#issuecomment-1243147966

   By this adjustment, the building speed with `-T 1C` parameter on `mvn` command is improved from about 2:50min to about 2min on my Mac book.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] FrankChen021 merged pull request #12501: Move web-console dependency declaration from druid-server to druid-distribution

Posted by GitBox <gi...@apache.org>.
FrankChen021 merged PR #12501:
URL: https://github.com/apache/druid/pull/12501


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org