You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by revans2 <gi...@git.apache.org> on 2016/11/27 03:48:05 UTC
[GitHub] storm pull request #1800: STORM-2217: Make DRPC pure java
GitHub user revans2 opened a pull request:
https://github.com/apache/storm/pull/1800
STORM-2217: Make DRPC pure java
This sets up some things for using Jersey for the UI, logviewer, and DRPC. It is not perfect and will probably need some clean up over time as we try to pull in UI and logviewer changes.
It divides up some of the functionality in DRPC so we could pull DRPC server and the webapp pieces into a separate module to separate out the classpaths. I have not shaded Jersey because of this, but am happy to try if people want me to.
I rewrote a lot of the core DRPC functionality so that when we do go to a newer version of jetty that supports async we can make DRPC much more scaleable.
I would love feedback on what I have done
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/revans2/incubator-storm STORM-2217
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/1800.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1800
----
commit ba2455da6353ecde84a66a91b1d5175d1922d4fb
Author: Robert (Bobby) Evans <ev...@yahoo-inc.com>
Date: 2016-11-24T20:43:31Z
STORM-2217: Make DRPC pure java with Jersy and prepare for separating classpaths.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
So shading is not really an option for jersey.
I was able to split the DRPC server off into its own package with tests, but packaging it up with the assembly plugin is proving to be difficult. If I upgrade to 3.0.0 of the assembly plugin I can use moduleSets and make it work, but I have to change {{storm-dist/binary}} to a multi-module build and move the code that actually packages the final release to a sub package under it. Oh and we leak two empty/useless jar files into the release package that should be ignored. moduleSets do not package pom modules. They require an artifact that is a file.
I really dislike all of these options. I see a few cleaner options, but they will require a lot more work.
1. Move to gradle
2. Change how we do shading so that the assembly subModule code works the way they intended it.
This would involve essentially having a separate package/build for what we want shaded (i.e. storm-shaded-deps.jar)
If someone has a cleaner solution I am happy to do it, but I think option 2 is the best so far, although I don't like it all that much.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
@ptgoetz if this looks good as is I will look into shading Jersey.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
Rebased the code
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1800
Okay, that last commit answered a lot of questions I had regarding packaging. ;)
+1 I was able to build a distribution, unpack it, and run the drpc service.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request #1800: STORM-2217: Make DRPC pure java
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/storm/pull/1800
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
@ptgoetz I have updated the packaging to have a separate directory for the DRPC server dependencies. I have run manual tests and everything works. The big difference is that {{apache-storm-2.0.0-SNAPSHOT.tar.gz}} and {{apache-storm-2.0.0-SNAPSHOT.zip}} are now under {{./final-package/target/}} instead of {{./target}} I will try and look to see if there are any docs that I need to update around this, but I wanted to give everyone a chance to look and give feedback on it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:
https://github.com/apache/storm/pull/1800
+1 from me. All manual tests are OK: build, dist packaging, unpacking, running drpc service.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1800
+1. Looks good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
Rebased again
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
One more option, still ugly, but with a lot less impact. I can do something similar to Hadoop. They use several different invocations of the maven assembly plugin into directories (predates moduleSets) and then have a separate script that produces the final release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by ptgoetz <gi...@git.apache.org>.
Github user ptgoetz commented on the issue:
https://github.com/apache/storm/pull/1800
@revans2 Okay. Let me review the packaging changes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
Shading Jersey is becoming rather difficult (lots of dependencies including aop and dependency injection. Splitting the DPRC server off into it's own location seems much simpler and less error prone, so I will spend some time on that instead.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm issue #1800: STORM-2217: Make DRPC pure java
Posted by revans2 <gi...@git.apache.org>.
Github user revans2 commented on the issue:
https://github.com/apache/storm/pull/1800
Will rebase soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---