You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Aaron Bossert <aa...@punchcyber.com> on 2019/01/10 15:58:13 UTC
support Hadoop 3.X
As requested by Thomas, I am hoping we can kick off a discussion about a
short-term goal of adding support for Hadoop 3.X, though, if I am not
mistaken, there was a pretty large change from 3.0 to 3.1...but will
confirm, in which case, the ask is to support 3.1+ specifically.
I program in Scala for the most part, but have not problem switching to
pure Java as needed. That being said, I am very unfamiliar with the Apex
code base and also unsure of the scope and level of effort required to get
this done. I look to the group for a few pointers on where to begin as
well as code style and such...I don't think I will need a lot of
hand-holding, just a good shove in the right direction...
Thanks again for the discussion and support on this...for background here
is my personal and professional motivation for pressing forward:
- Doing research work for clients as well as internal R&D for streaming
data processing
- very specific requirement to support the ability to alter
topologies programatically and manually on the fly, hence the interest in
Apex, which is the only platform I know of that has this support built
in...I don't want to re-invent that wheel unless absolutely necessary
- General interest in supporting the open-source community, both as a
company and individually
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
It was an issue in the chair to keyboard interface...just submitted PR 607
<https://github.com/apache/apex-core/pull/607>. The Jenkins
build/test/whatever is still running...
On Tue, Jan 15, 2019 at 11:36 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> Shouldn't require additional authorization unless something changed
> recently. Did you try creating a PR from your branch in your fork on
> github.com?
>
> Thanks
>
> On Tue, Jan 15, 2019 at 6:59 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
> > When I try to do a pull request from Intellij, I get a permissions
> > error...I have not done this much, so it could be something really dumb
> on
> > my part...do I need you to authorize my account in some way?
> >
> > On Tue, Jan 15, 2019 at 3:20 PM Aaron Bossert <aa...@punchcyber.com>
> > wrote:
> >
> > > OK, I have my local clone working and updated for Hadoop 3.1...
> > >
> > > On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com>
> > > wrote:
> > >
> > >> OK, folks. I have imported the Master branch of apex-core....I
> updated
> > >> the dependency for Hadoop to 3.1.0 and here is first round of
> > problems. Of
> > >> those listed below, all but the last two constants have equivalents in
> > the
> > >> newer version that were easy enough to do a find and replace
> > >> with...however, the last two are shown in 2.6.1 as deprecated, but no
> > >> mention of a replacement...Perhaps someone more familiar with the Apex
> > code
> > >> base can chime in on these two constants (highlighted in red-ish
> toward
> > the
> > >> bottom and copied just below for convenience)?
> > >>
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> > >> cannot find symbol
> > >> symbol: variable HADOOP_SSL_ENABLED_KEY
> > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> > >> cannot find symbol
> > >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> > >>
> > >>
> > >>
> > >> [INFO] -------------------------------------------------------------
> > >> [WARNING] COMPILATION WARNING :
> > >> [INFO] -------------------------------------------------------------
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16]
> > >> sun.misc.Signal is internal proprietary API and may be removed in a
> > future
> > >> release
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16]
> > >> sun.misc.SignalHandler is internal proprietary API and may be removed
> > in a
> > >> future release
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27]
> > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> > >> com.datatorrent.api.LocalMode in com.datatorrent.api has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19]
> > >>
> >
> <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T)
> > >> in com.datatorrent.api.DAG has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17]
> > >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11]
> > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> > >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21]
> > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats)
> in
> > >> com.datatorrent.api.StatsListener has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47]
> > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats)
> in
> > >> com.datatorrent.api.StatsListener has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21]
> > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats)
> in
> > >> com.datatorrent.api.StatsListener has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22]
> > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats)
> in
> > >> com.datatorrent.api.StatsListener has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22]
> > >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats)
> in
> > >> com.datatorrent.api.StatsListener has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22]
> > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18]
> > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35]
> > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> > >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69]
> > >> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42]
> > >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> > >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62]
> > >> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has
> > been
> > >> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44]
> > >> counters in com.datatorrent.api.Stats.OperatorStats has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74]
> > >> counters in com.datatorrent.api.Stats.OperatorStats has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72]
> > >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43]
> > >> com.datatorrent.api.Operator.CheckpointListener in
> > >> com.datatorrent.api.Operator has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23]
> > >> com.datatorrent.api.Operator.CheckpointListener in
> > >> com.datatorrent.api.Operator has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37]
> > >> com.datatorrent.api.Operator.CheckpointListener in
> > >> com.datatorrent.api.Operator has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17]
> > >> com.datatorrent.api.Operator.CheckpointListener in
> > >> com.datatorrent.api.Operator has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66]
> > >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> > deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10]
> > >> counters in com.datatorrent.api.Stats.OperatorStats has been
> deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26]
> > >> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9]
> > >> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20]
> > >> writeStringToFile(java.io.File,java.lang.String,boolean) in
> > >> org.apache.commons.io.FileUtils has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16]
> > >> copy(java.io.InputStream,java.io.Writer) in
> > org.apache.commons.io.IOUtils
> > >> has been deprecated
> > >> [WARNING]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21]
> > >> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has
> > been
> > >> deprecated
> > >> [INFO] 100 warnings
> > >> [INFO] -------------------------------------------------------------
> > >> [INFO] -------------------------------------------------------------
> > >> [ERROR] COMPILATION ERROR :
> > >> [INFO] -------------------------------------------------------------
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178]
> > >> cannot find symbol
> > >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> > >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> > >> cannot find symbol
> > >> symbol: variable HADOOP_SSL_ENABLED_KEY
> > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> > >> [ERROR]
> > >>
> >
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> > >> cannot find symbol
> > >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> > >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> > >>
> > >> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com>
> > >> wrote:
> > >>
> > >>> Thanks!
> > >>>
> > >>> Sent from my iPhone
> > >>>
> > >>> > On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
> > >>> >
> > >>> > For the upgrade it is probably best when you change the version
> > number
> > >>> in
> > >>> > [1] and then work through the build issues.
> > >>> >
> > >>> > I would expect a number of dependency changes since the client
> > >>> classpath
> > >>> > now has the Hadoop transitive dependencies shaded away [2].
> > >>> >
> > >>> > Regarding the development process, see [3].
> > >>> >
> > >>> > Thomas
> > >>> >
> > >>> > [1]
> > >>> >
> > >>>
> >
> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
> > >>> > [2] https://hadoop.apache.org/docs/r3.0.0/
> > >>> > [3] http://apex.apache.org/contributing.html
> > >>> >
> > >>> >
> > >>> >
> > >>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <
> aaron@punchcyber.com
> > >
> > >>> wrote:
> > >>> >>
> > >>> >> at first blush, I would think, that the YARN node tagging and
> native
> > >>> Docker
> > >>> >> support would be particularly intriguing...the GPU node support
> > would
> > >>> be a
> > >>> >> good implementation trick to be able to put operators that do deep
> > >>> learning
> > >>> >> tasks...
> > >>> >>
> > >>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <
> > >>> pramod.immaneni@gmail.com
> > >>> >>>
> > >>> >> wrote:
> > >>> >>
> > >>> >>> I agree we should just try changing the dependency first. I am in
> > >>> favor
> > >>> >> of
> > >>> >>> upgrading. It will be good to explore if and how we can leverage
> > >>> some of
> > >>> >>> the new features in 3.x.
> > >>> >>>
> > >>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org>
> > >>> wrote:
> > >>> >>>>
> > >>> >>>> Hi Aaron,
> > >>> >>>>
> > >>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
> > >>> >>>> recompiling and running unit test. Also, before you put
> > significant
> > >>> >>> effort
> > >>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex.
> Once
> > >>> >>>> everyone agrees and vote passes, please open JIRA for tracking
> the
> > >>> >>> changes.
> > >>> >>>>
> > >>> >>>> Thank you,
> > >>> >>>>
> > >>> >>>> Vlad
> > >>> >>>>
> > >>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aaron@punchcyber.com
> >
> > >>> >> wrote:
> > >>> >>>>>
> > >>> >>>>> As requested by Thomas, I am hoping we can kick off a
> discussion
> > >>> >> about
> > >>> >>> a
> > >>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I
> am
> > >>> not
> > >>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but
> > >>> will
> > >>> >>>>> confirm, in which case, the ask is to support 3.1+
> specifically.
> > >>> >>>>>
> > >>> >>>>> I program in Scala for the most part, but have not problem
> > >>> switching
> > >>> >> to
> > >>> >>>>> pure Java as needed. That being said, I am very unfamiliar
> with
> > >>> the
> > >>> >>> Apex
> > >>> >>>>> code base and also unsure of the scope and level of effort
> > required
> > >>> >> to
> > >>> >>>> get
> > >>> >>>>> this done. I look to the group for a few pointers on where to
> > >>> begin
> > >>> >> as
> > >>> >>>>> well as code style and such...I don't think I will need a lot
> of
> > >>> >>>>> hand-holding, just a good shove in the right direction...
> > >>> >>>>>
> > >>> >>>>> Thanks again for the discussion and support on this...for
> > >>> background
> > >>> >>> here
> > >>> >>>>> is my personal and professional motivation for pressing
> forward:
> > >>> >>>>>
> > >>> >>>>> - Doing research work for clients as well as internal R&D for
> > >>> >>> streaming
> > >>> >>>>> data processing
> > >>> >>>>> - very specific requirement to support the ability to alter
> > >>> >>>>> topologies programatically and manually on the fly, hence
> the
> > >>> >>>> interest in
> > >>> >>>>> Apex, which is the only platform I know of that has this
> > >>> support
> > >>> >>>> built
> > >>> >>>>> in...I don't want to re-invent that wheel unless absolutely
> > >>> >>>> necessary
> > >>> >>>>> - General interest in supporting the open-source community,
> both
> > >>> >> as a
> > >>> >>>>> company and individually
> > >>> >>>>>
> > >>> >>>>> --
> > >>> >>>>>
> > >>> >>>>> M. Aaron Bossert
> > >>> >>>>> (202) 594-1703
> > >>> >>>>> Punch Cyber Analytics Group
> > >>> >>>>
> > >>> >>>> --
> > >>> >>> Thanks,
> > >>> >>> Pramod
> > >>> >>> http://ts.la/pramod3443
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>> >> --
> > >>> >>
> > >>> >> M. Aaron Bossert
> > >>> >> (202) 594-1703
> > >>> >> Punch Cyber Analytics Group
> > >>> >>
> > >>>
> > >>
> > >>
> > >> --
> > >>
> > >> M. Aaron Bossert
> > >> (202) 594-1703
> > >> Punch Cyber Analytics Group
> > >>
> > >>
> > >>
> > >
> > > --
> > >
> > > M. Aaron Bossert
> > > (202) 594-1703
> > > Punch Cyber Analytics Group
> > >
> > >
> > >
> >
> > --
> >
> > M. Aaron Bossert
> > (202) 594-1703
> > Punch Cyber Analytics Group
> >
>
>
> --
> Thanks,
> Pramod
> http://ts.la/pramod3443
>
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Pramod Immaneni <pr...@gmail.com>.
Shouldn't require additional authorization unless something changed
recently. Did you try creating a PR from your branch in your fork on
github.com?
Thanks
On Tue, Jan 15, 2019 at 6:59 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> When I try to do a pull request from Intellij, I get a permissions
> error...I have not done this much, so it could be something really dumb on
> my part...do I need you to authorize my account in some way?
>
> On Tue, Jan 15, 2019 at 3:20 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
> > OK, I have my local clone working and updated for Hadoop 3.1...
> >
> > On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com>
> > wrote:
> >
> >> OK, folks. I have imported the Master branch of apex-core....I updated
> >> the dependency for Hadoop to 3.1.0 and here is first round of
> problems. Of
> >> those listed below, all but the last two constants have equivalents in
> the
> >> newer version that were easy enough to do a find and replace
> >> with...however, the last two are shown in 2.6.1 as deprecated, but no
> >> mention of a replacement...Perhaps someone more familiar with the Apex
> code
> >> base can chime in on these two constants (highlighted in red-ish toward
> the
> >> bottom and copied just below for convenience)?
> >>
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> >> cannot find symbol
> >> symbol: variable HADOOP_SSL_ENABLED_KEY
> >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> >> cannot find symbol
> >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> >>
> >>
> >>
> >> [INFO] -------------------------------------------------------------
> >> [WARNING] COMPILATION WARNING :
> >> [INFO] -------------------------------------------------------------
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16]
> >> sun.misc.Signal is internal proprietary API and may be removed in a
> future
> >> release
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16]
> >> sun.misc.SignalHandler is internal proprietary API and may be removed
> in a
> >> future release
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27]
> >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> >> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19]
> >>
> <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T)
> >> in com.datatorrent.api.DAG has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17]
> >> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11]
> >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21]
> >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> >> com.datatorrent.api.StatsListener has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47]
> >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> >> com.datatorrent.api.StatsListener has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21]
> >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> >> com.datatorrent.api.StatsListener has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22]
> >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> >> com.datatorrent.api.StatsListener has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22]
> >> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> >> com.datatorrent.api.StatsListener has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22]
> >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18]
> >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35]
> >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69]
> >> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42]
> >> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> >> com.datatorrent.stram.util.ConfigUtils has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62]
> >> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has
> been
> >> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44]
> >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74]
> >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72]
> >> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43]
> >> com.datatorrent.api.Operator.CheckpointListener in
> >> com.datatorrent.api.Operator has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23]
> >> com.datatorrent.api.Operator.CheckpointListener in
> >> com.datatorrent.api.Operator has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37]
> >> com.datatorrent.api.Operator.CheckpointListener in
> >> com.datatorrent.api.Operator has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17]
> >> com.datatorrent.api.Operator.CheckpointListener in
> >> com.datatorrent.api.Operator has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66]
> >> com.datatorrent.api.StatsListener in com.datatorrent.api has been
> deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10]
> >> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26]
> >> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9]
> >> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20]
> >> writeStringToFile(java.io.File,java.lang.String,boolean) in
> >> org.apache.commons.io.FileUtils has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16]
> >> copy(java.io.InputStream,java.io.Writer) in
> org.apache.commons.io.IOUtils
> >> has been deprecated
> >> [WARNING]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21]
> >> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has
> been
> >> deprecated
> >> [INFO] 100 warnings
> >> [INFO] -------------------------------------------------------------
> >> [INFO] -------------------------------------------------------------
> >> [ERROR] COMPILATION ERROR :
> >> [INFO] -------------------------------------------------------------
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178]
> >> cannot find symbol
> >> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> >> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> >> cannot find symbol
> >> symbol: variable HADOOP_SSL_ENABLED_KEY
> >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> >> [ERROR]
> >>
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> >> cannot find symbol
> >> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> >> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> >>
> >> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com>
> >> wrote:
> >>
> >>> Thanks!
> >>>
> >>> Sent from my iPhone
> >>>
> >>> > On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
> >>> >
> >>> > For the upgrade it is probably best when you change the version
> number
> >>> in
> >>> > [1] and then work through the build issues.
> >>> >
> >>> > I would expect a number of dependency changes since the client
> >>> classpath
> >>> > now has the Hadoop transitive dependencies shaded away [2].
> >>> >
> >>> > Regarding the development process, see [3].
> >>> >
> >>> > Thomas
> >>> >
> >>> > [1]
> >>> >
> >>>
> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
> >>> > [2] https://hadoop.apache.org/docs/r3.0.0/
> >>> > [3] http://apex.apache.org/contributing.html
> >>> >
> >>> >
> >>> >
> >>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aaron@punchcyber.com
> >
> >>> wrote:
> >>> >>
> >>> >> at first blush, I would think, that the YARN node tagging and native
> >>> Docker
> >>> >> support would be particularly intriguing...the GPU node support
> would
> >>> be a
> >>> >> good implementation trick to be able to put operators that do deep
> >>> learning
> >>> >> tasks...
> >>> >>
> >>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <
> >>> pramod.immaneni@gmail.com
> >>> >>>
> >>> >> wrote:
> >>> >>
> >>> >>> I agree we should just try changing the dependency first. I am in
> >>> favor
> >>> >> of
> >>> >>> upgrading. It will be good to explore if and how we can leverage
> >>> some of
> >>> >>> the new features in 3.x.
> >>> >>>
> >>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org>
> >>> wrote:
> >>> >>>>
> >>> >>>> Hi Aaron,
> >>> >>>>
> >>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
> >>> >>>> recompiling and running unit test. Also, before you put
> significant
> >>> >>> effort
> >>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
> >>> >>>> everyone agrees and vote passes, please open JIRA for tracking the
> >>> >>> changes.
> >>> >>>>
> >>> >>>> Thank you,
> >>> >>>>
> >>> >>>> Vlad
> >>> >>>>
> >>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
> >>> >> wrote:
> >>> >>>>>
> >>> >>>>> As requested by Thomas, I am hoping we can kick off a discussion
> >>> >> about
> >>> >>> a
> >>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am
> >>> not
> >>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but
> >>> will
> >>> >>>>> confirm, in which case, the ask is to support 3.1+ specifically.
> >>> >>>>>
> >>> >>>>> I program in Scala for the most part, but have not problem
> >>> switching
> >>> >> to
> >>> >>>>> pure Java as needed. That being said, I am very unfamiliar with
> >>> the
> >>> >>> Apex
> >>> >>>>> code base and also unsure of the scope and level of effort
> required
> >>> >> to
> >>> >>>> get
> >>> >>>>> this done. I look to the group for a few pointers on where to
> >>> begin
> >>> >> as
> >>> >>>>> well as code style and such...I don't think I will need a lot of
> >>> >>>>> hand-holding, just a good shove in the right direction...
> >>> >>>>>
> >>> >>>>> Thanks again for the discussion and support on this...for
> >>> background
> >>> >>> here
> >>> >>>>> is my personal and professional motivation for pressing forward:
> >>> >>>>>
> >>> >>>>> - Doing research work for clients as well as internal R&D for
> >>> >>> streaming
> >>> >>>>> data processing
> >>> >>>>> - very specific requirement to support the ability to alter
> >>> >>>>> topologies programatically and manually on the fly, hence the
> >>> >>>> interest in
> >>> >>>>> Apex, which is the only platform I know of that has this
> >>> support
> >>> >>>> built
> >>> >>>>> in...I don't want to re-invent that wheel unless absolutely
> >>> >>>> necessary
> >>> >>>>> - General interest in supporting the open-source community, both
> >>> >> as a
> >>> >>>>> company and individually
> >>> >>>>>
> >>> >>>>> --
> >>> >>>>>
> >>> >>>>> M. Aaron Bossert
> >>> >>>>> (202) 594-1703
> >>> >>>>> Punch Cyber Analytics Group
> >>> >>>>
> >>> >>>> --
> >>> >>> Thanks,
> >>> >>> Pramod
> >>> >>> http://ts.la/pramod3443
> >>> >>>
> >>> >>
> >>> >>
> >>> >> --
> >>> >>
> >>> >> M. Aaron Bossert
> >>> >> (202) 594-1703
> >>> >> Punch Cyber Analytics Group
> >>> >>
> >>>
> >>
> >>
> >> --
> >>
> >> M. Aaron Bossert
> >> (202) 594-1703
> >> Punch Cyber Analytics Group
> >>
> >>
> >>
> >
> > --
> >
> > M. Aaron Bossert
> > (202) 594-1703
> > Punch Cyber Analytics Group
> >
> >
> >
>
> --
>
> M. Aaron Bossert
> (202) 594-1703
> Punch Cyber Analytics Group
>
--
Thanks,
Pramod
http://ts.la/pramod3443
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
When I try to do a pull request from Intellij, I get a permissions
error...I have not done this much, so it could be something really dumb on
my part...do I need you to authorize my account in some way?
On Tue, Jan 15, 2019 at 3:20 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> OK, I have my local clone working and updated for Hadoop 3.1...
>
> On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> OK, folks. I have imported the Master branch of apex-core....I updated
>> the dependency for Hadoop to 3.1.0 and here is first round of problems. Of
>> those listed below, all but the last two constants have equivalents in the
>> newer version that were easy enough to do a find and replace
>> with...however, the last two are shown in 2.6.1 as deprecated, but no
>> mention of a replacement...Perhaps someone more familiar with the Apex code
>> base can chime in on these two constants (highlighted in red-ish toward the
>> bottom and copied just below for convenience)?
>>
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
>> cannot find symbol
>> symbol: variable HADOOP_SSL_ENABLED_KEY
>> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
>> cannot find symbol
>> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
>> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>>
>>
>>
>> [INFO] -------------------------------------------------------------
>> [WARNING] COMPILATION WARNING :
>> [INFO] -------------------------------------------------------------
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16]
>> sun.misc.Signal is internal proprietary API and may be removed in a future
>> release
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16]
>> sun.misc.SignalHandler is internal proprietary API and may be removed in a
>> future release
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27]
>> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
>> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
>> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19]
>> <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T)
>> in com.datatorrent.api.DAG has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17]
>> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11]
>> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
>> com.datatorrent.stram.util.ConfigUtils has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21]
>> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
>> com.datatorrent.api.StatsListener has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47]
>> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
>> com.datatorrent.api.StatsListener has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21]
>> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
>> com.datatorrent.api.StatsListener has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22]
>> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
>> com.datatorrent.api.StatsListener has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22]
>> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
>> com.datatorrent.api.StatsListener has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22]
>> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18]
>> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35]
>> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
>> com.datatorrent.stram.util.ConfigUtils has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69]
>> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42]
>> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
>> com.datatorrent.stram.util.ConfigUtils has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62]
>> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been
>> deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44]
>> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74]
>> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72]
>> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43]
>> com.datatorrent.api.Operator.CheckpointListener in
>> com.datatorrent.api.Operator has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23]
>> com.datatorrent.api.Operator.CheckpointListener in
>> com.datatorrent.api.Operator has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37]
>> com.datatorrent.api.Operator.CheckpointListener in
>> com.datatorrent.api.Operator has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17]
>> com.datatorrent.api.Operator.CheckpointListener in
>> com.datatorrent.api.Operator has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66]
>> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10]
>> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26]
>> counters in com.datatorrent.stram.api.BaseContext has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9]
>> counters in com.datatorrent.stram.api.BaseContext has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20]
>> writeStringToFile(java.io.File,java.lang.String,boolean) in
>> org.apache.commons.io.FileUtils has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16]
>> copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils
>> has been deprecated
>> [WARNING]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21]
>> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been
>> deprecated
>> [INFO] 100 warnings
>> [INFO] -------------------------------------------------------------
>> [INFO] -------------------------------------------------------------
>> [ERROR] COMPILATION ERROR :
>> [INFO] -------------------------------------------------------------
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178]
>> cannot find symbol
>> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
>> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
>> cannot find symbol
>> symbol: variable HADOOP_SSL_ENABLED_KEY
>> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>> [ERROR]
>> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
>> cannot find symbol
>> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
>> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>>
>> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>
>>> Thanks!
>>>
>>> Sent from my iPhone
>>>
>>> > On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
>>> >
>>> > For the upgrade it is probably best when you change the version number
>>> in
>>> > [1] and then work through the build issues.
>>> >
>>> > I would expect a number of dependency changes since the client
>>> classpath
>>> > now has the Hadoop transitive dependencies shaded away [2].
>>> >
>>> > Regarding the development process, see [3].
>>> >
>>> > Thomas
>>> >
>>> > [1]
>>> >
>>> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
>>> > [2] https://hadoop.apache.org/docs/r3.0.0/
>>> > [3] http://apex.apache.org/contributing.html
>>> >
>>> >
>>> >
>>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com>
>>> wrote:
>>> >>
>>> >> at first blush, I would think, that the YARN node tagging and native
>>> Docker
>>> >> support would be particularly intriguing...the GPU node support would
>>> be a
>>> >> good implementation trick to be able to put operators that do deep
>>> learning
>>> >> tasks...
>>> >>
>>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <
>>> pramod.immaneni@gmail.com
>>> >>>
>>> >> wrote:
>>> >>
>>> >>> I agree we should just try changing the dependency first. I am in
>>> favor
>>> >> of
>>> >>> upgrading. It will be good to explore if and how we can leverage
>>> some of
>>> >>> the new features in 3.x.
>>> >>>
>>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org>
>>> wrote:
>>> >>>>
>>> >>>> Hi Aaron,
>>> >>>>
>>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
>>> >>>> recompiling and running unit test. Also, before you put significant
>>> >>> effort
>>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
>>> >>>> everyone agrees and vote passes, please open JIRA for tracking the
>>> >>> changes.
>>> >>>>
>>> >>>> Thank you,
>>> >>>>
>>> >>>> Vlad
>>> >>>>
>>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
>>> >> wrote:
>>> >>>>>
>>> >>>>> As requested by Thomas, I am hoping we can kick off a discussion
>>> >> about
>>> >>> a
>>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am
>>> not
>>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but
>>> will
>>> >>>>> confirm, in which case, the ask is to support 3.1+ specifically.
>>> >>>>>
>>> >>>>> I program in Scala for the most part, but have not problem
>>> switching
>>> >> to
>>> >>>>> pure Java as needed. That being said, I am very unfamiliar with
>>> the
>>> >>> Apex
>>> >>>>> code base and also unsure of the scope and level of effort required
>>> >> to
>>> >>>> get
>>> >>>>> this done. I look to the group for a few pointers on where to
>>> begin
>>> >> as
>>> >>>>> well as code style and such...I don't think I will need a lot of
>>> >>>>> hand-holding, just a good shove in the right direction...
>>> >>>>>
>>> >>>>> Thanks again for the discussion and support on this...for
>>> background
>>> >>> here
>>> >>>>> is my personal and professional motivation for pressing forward:
>>> >>>>>
>>> >>>>> - Doing research work for clients as well as internal R&D for
>>> >>> streaming
>>> >>>>> data processing
>>> >>>>> - very specific requirement to support the ability to alter
>>> >>>>> topologies programatically and manually on the fly, hence the
>>> >>>> interest in
>>> >>>>> Apex, which is the only platform I know of that has this
>>> support
>>> >>>> built
>>> >>>>> in...I don't want to re-invent that wheel unless absolutely
>>> >>>> necessary
>>> >>>>> - General interest in supporting the open-source community, both
>>> >> as a
>>> >>>>> company and individually
>>> >>>>>
>>> >>>>> --
>>> >>>>>
>>> >>>>> M. Aaron Bossert
>>> >>>>> (202) 594-1703
>>> >>>>> Punch Cyber Analytics Group
>>> >>>>
>>> >>>> --
>>> >>> Thanks,
>>> >>> Pramod
>>> >>> http://ts.la/pramod3443
>>> >>>
>>> >>
>>> >>
>>> >> --
>>> >>
>>> >> M. Aaron Bossert
>>> >> (202) 594-1703
>>> >> Punch Cyber Analytics Group
>>> >>
>>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (202) 594-1703
>> Punch Cyber Analytics Group
>>
>>
>>
>
> --
>
> M. Aaron Bossert
> (202) 594-1703
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
OK, I have my local clone working and updated for Hadoop 3.1...
On Tue, Jan 15, 2019 at 1:55 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> OK, folks. I have imported the Master branch of apex-core....I updated
> the dependency for Hadoop to 3.1.0 and here is first round of problems. Of
> those listed below, all but the last two constants have equivalents in the
> newer version that were easy enough to do a find and replace
> with...however, the last two are shown in 2.6.1 as deprecated, but no
> mention of a replacement...Perhaps someone more familiar with the Apex code
> base can chime in on these two constants (highlighted in red-ish toward the
> bottom and copied just below for convenience)?
>
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> cannot find symbol
> symbol: variable HADOOP_SSL_ENABLED_KEY
> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> cannot find symbol
> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>
>
>
> [INFO] -------------------------------------------------------------
> [WARNING] COMPILATION WARNING :
> [INFO] -------------------------------------------------------------
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16]
> sun.misc.Signal is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16]
> sun.misc.SignalHandler is internal proprietary API and may be removed in a
> future release
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27]
> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
> com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19]
> <T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T)
> in com.datatorrent.api.DAG has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17]
> setCounters(java.lang.Object) in com.datatorrent.api.Context has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11]
> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> com.datatorrent.stram.util.ConfigUtils has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21]
> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> com.datatorrent.api.StatsListener has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47]
> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> com.datatorrent.api.StatsListener has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21]
> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> com.datatorrent.api.StatsListener has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22]
> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> com.datatorrent.api.StatsListener has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22]
> processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
> com.datatorrent.api.StatsListener has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22]
> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18]
> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35]
> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> com.datatorrent.stram.util.ConfigUtils has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69]
> getMemory() in org.apache.hadoop.yarn.api.records.Resource has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42]
> getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
> com.datatorrent.stram.util.ConfigUtils has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62]
> COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been
> deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44]
> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74]
> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72]
> SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43]
> com.datatorrent.api.Operator.CheckpointListener in
> com.datatorrent.api.Operator has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23]
> com.datatorrent.api.Operator.CheckpointListener in
> com.datatorrent.api.Operator has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37]
> com.datatorrent.api.Operator.CheckpointListener in
> com.datatorrent.api.Operator has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17]
> com.datatorrent.api.Operator.CheckpointListener in
> com.datatorrent.api.Operator has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66]
> com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10]
> counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26]
> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9]
> counters in com.datatorrent.stram.api.BaseContext has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20]
> writeStringToFile(java.io.File,java.lang.String,boolean) in
> org.apache.commons.io.FileUtils has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16]
> copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils
> has been deprecated
> [WARNING]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21]
> readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been
> deprecated
> [INFO] 100 warnings
> [INFO] -------------------------------------------------------------
> [INFO] -------------------------------------------------------------
> [ERROR] COMPILATION ERROR :
> [INFO] -------------------------------------------------------------
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178]
> cannot find symbol
> symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
> location: class org.apache.hadoop.yarn.conf.YarnConfiguration
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
> cannot find symbol
> symbol: variable HADOOP_SSL_ENABLED_KEY
> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
> [ERROR]
> /Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
> cannot find symbol
> symbol: variable HADOOP_SSL_ENABLED_DEFAULT
> location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
>
> On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
>
>> Thanks!
>>
>> Sent from my iPhone
>>
>> > On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
>> >
>> > For the upgrade it is probably best when you change the version number
>> in
>> > [1] and then work through the build issues.
>> >
>> > I would expect a number of dependency changes since the client classpath
>> > now has the Hadoop transitive dependencies shaded away [2].
>> >
>> > Regarding the development process, see [3].
>> >
>> > Thomas
>> >
>> > [1]
>> >
>> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
>> > [2] https://hadoop.apache.org/docs/r3.0.0/
>> > [3] http://apex.apache.org/contributing.html
>> >
>> >
>> >
>> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>> >>
>> >> at first blush, I would think, that the YARN node tagging and native
>> Docker
>> >> support would be particularly intriguing...the GPU node support would
>> be a
>> >> good implementation trick to be able to put operators that do deep
>> learning
>> >> tasks...
>> >>
>> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <
>> pramod.immaneni@gmail.com
>> >>>
>> >> wrote:
>> >>
>> >>> I agree we should just try changing the dependency first. I am in
>> favor
>> >> of
>> >>> upgrading. It will be good to explore if and how we can leverage some
>> of
>> >>> the new features in 3.x.
>> >>>
>> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org>
>> wrote:
>> >>>>
>> >>>> Hi Aaron,
>> >>>>
>> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
>> >>>> recompiling and running unit test. Also, before you put significant
>> >>> effort
>> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
>> >>>> everyone agrees and vote passes, please open JIRA for tracking the
>> >>> changes.
>> >>>>
>> >>>> Thank you,
>> >>>>
>> >>>> Vlad
>> >>>>
>> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
>> >> wrote:
>> >>>>>
>> >>>>> As requested by Thomas, I am hoping we can kick off a discussion
>> >> about
>> >>> a
>> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am
>> not
>> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but will
>> >>>>> confirm, in which case, the ask is to support 3.1+ specifically.
>> >>>>>
>> >>>>> I program in Scala for the most part, but have not problem switching
>> >> to
>> >>>>> pure Java as needed. That being said, I am very unfamiliar with the
>> >>> Apex
>> >>>>> code base and also unsure of the scope and level of effort required
>> >> to
>> >>>> get
>> >>>>> this done. I look to the group for a few pointers on where to begin
>> >> as
>> >>>>> well as code style and such...I don't think I will need a lot of
>> >>>>> hand-holding, just a good shove in the right direction...
>> >>>>>
>> >>>>> Thanks again for the discussion and support on this...for background
>> >>> here
>> >>>>> is my personal and professional motivation for pressing forward:
>> >>>>>
>> >>>>> - Doing research work for clients as well as internal R&D for
>> >>> streaming
>> >>>>> data processing
>> >>>>> - very specific requirement to support the ability to alter
>> >>>>> topologies programatically and manually on the fly, hence the
>> >>>> interest in
>> >>>>> Apex, which is the only platform I know of that has this support
>> >>>> built
>> >>>>> in...I don't want to re-invent that wheel unless absolutely
>> >>>> necessary
>> >>>>> - General interest in supporting the open-source community, both
>> >> as a
>> >>>>> company and individually
>> >>>>>
>> >>>>> --
>> >>>>>
>> >>>>> M. Aaron Bossert
>> >>>>> (202) 594-1703
>> >>>>> Punch Cyber Analytics Group
>> >>>>
>> >>>> --
>> >>> Thanks,
>> >>> Pramod
>> >>> http://ts.la/pramod3443
>> >>>
>> >>
>> >>
>> >> --
>> >>
>> >> M. Aaron Bossert
>> >> (202) 594-1703
>> >> Punch Cyber Analytics Group
>> >>
>>
>
>
> --
>
> M. Aaron Bossert
> (202) 594-1703
> Punch Cyber Analytics Group
>
>
>
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
OK, folks. I have imported the Master branch of apex-core....I updated the
dependency for Hadoop to 3.1.0 and here is first round of problems. Of
those listed below, all but the last two constants have equivalents in the
newer version that were easy enough to do a find and replace
with...however, the last two are shown in 2.6.1 as deprecated, but no
mention of a replacement...Perhaps someone more familiar with the Apex code
base can chime in on these two constants (highlighted in red-ish toward the
bottom and copied just below for convenience)?
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
cannot find symbol
symbol: variable HADOOP_SSL_ENABLED_KEY
location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
cannot find symbol
symbol: variable HADOOP_SSL_ENABLED_DEFAULT
location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING :
[INFO] -------------------------------------------------------------
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[120,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[37,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[38,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[44,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[29,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[30,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[67,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[68,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[69,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[70,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/RequestFactory.java:[26,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[70,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[71,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[33,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[146,16]
sun.misc.Signal is internal proprietary API and may be removed in a future
release
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/cli/ApexCli.java:[147,16]
sun.misc.SignalHandler is internal proprietary API and may be removed in a
future release
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/EmbeddedAppLauncherImpl.java:[32,27]
com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StramLocalCluster.java:[45,27]
com.datatorrent.api.LocalMode in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[63,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/StreamingContainer.java:[64,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeGetPropertyRequest.java:[33,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/api/plugin/DAGExecutionPlugin.java:[29,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/org/apache/apex/engine/plugin/AbstractDAGExecutionPluginContext.java:[32,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StramToNodeSetPropertyRequest.java:[30,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Slider.java:[31,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorResponse.java:[23,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[41,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/debug/TupleRecorderCollection.java:[42,27]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[78,15]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[199,17]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/OperatorDeployInfo.java:[270,17]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/BaseContext.java:[70,15]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/api/StreamingContainerUmbilicalProtocol.java:[188,22]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[1523,19]
<T>setAttribute(com.datatorrent.api.Operator,com.datatorrent.api.Attribute<T>,T)
in com.datatorrent.api.DAG has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[230,15]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[309,17]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[453,17]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[925,17]
setCounters(java.lang.Object) in com.datatorrent.api.Context has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PTOperator.java:[258,25]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerAgent.java:[465,11]
getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
com.datatorrent.stram.util.ConfigUtils has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[85,38]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[86,15]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/OperatorStatus.java:[160,15]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[210,21]
processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
com.datatorrent.api.StatsListener has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,16]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[212,47]
processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
com.datatorrent.api.StatsListener has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,7]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[218,33]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[231,73]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[242,21]
processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
com.datatorrent.api.StatsListener has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[234,19]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[236,50]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[244,22]
processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
com.datatorrent.api.StatsListener has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[250,22]
processStats(com.datatorrent.api.StatsListener.BatchedOperatorStats) in
com.datatorrent.api.StatsListener has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[254,67]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[815,16]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[820,12]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[825,52]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/physical/PhysicalPlan.java:[1845,13]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[418,22]
SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[438,18]
SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[492,35]
getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
com.datatorrent.stram.util.ConfigUtils has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[497,69]
getMemory() in org.apache.hadoop.yarn.api.records.Resource has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[508,42]
getSchemePrefix(org.apache.hadoop.yarn.conf.YarnConfiguration) in
com.datatorrent.stram.util.ConfigUtils has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1048,9]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1557,14]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1720,62]
COUNTERS_AGGREGATOR in com.datatorrent.api.Context.OperatorContext has been
deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[1721,44]
counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[2593,74]
counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/StreamingContainerManager.java:[3142,72]
SystemClock() in org.apache.hadoop.yarn.util.SystemClock has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[129,30]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[356,16]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[409,7]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[485,43]
com.datatorrent.api.Operator.CheckpointListener in
com.datatorrent.api.Operator has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[486,23]
com.datatorrent.api.Operator.CheckpointListener in
com.datatorrent.api.Operator has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[572,37]
com.datatorrent.api.Operator.CheckpointListener in
com.datatorrent.api.Operator has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[573,17]
com.datatorrent.api.Operator.CheckpointListener in
com.datatorrent.api.Operator has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[636,16]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[657,16]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[659,75]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[666,51]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/Node.java:[667,66]
com.datatorrent.api.StatsListener in com.datatorrent.api has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,10]
counters in com.datatorrent.api.Stats.OperatorStats has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[147,26]
counters in com.datatorrent.stram.api.BaseContext has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/engine/OperatorContext.java:[148,9]
counters in com.datatorrent.stram.api.BaseContext has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[171,20]
writeStringToFile(java.io.File,java.lang.String,boolean) in
org.apache.commons.io.FileUtils has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[193,16]
copy(java.io.InputStream,java.io.Writer) in org.apache.commons.io.IOUtils
has been deprecated
[WARNING]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/ClassPathResolvers.java:[198,21]
readFileToString(java.io.File) in org.apache.commons.io.FileUtils has been
deprecated
[INFO] 100 warnings
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[166,94]
cannot find symbol
symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/plan/logical/LogicalPlan.java:[167,87]
cannot find symbol
symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,120]
cannot find symbol
symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_KEY
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[615,173]
cannot find symbol
symbol: variable DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,123]
cannot find symbol
symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_KEY
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramAppLauncher.java:[618,178]
cannot find symbol
symbol: variable DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT
location: class org.apache.hadoop.yarn.conf.YarnConfiguration
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,71]
cannot find symbol
symbol: variable HADOOP_SSL_ENABLED_KEY
location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
[ERROR]
/Users/mbossert/IdeaProjects/apex-core/engine/src/main/java/com/datatorrent/stram/client/StramClientUtils.java:[831,125]
cannot find symbol
symbol: variable HADOOP_SSL_ENABLED_DEFAULT
location: class org.apache.hadoop.fs.CommonConfigurationKeysPublic
On Sun, Jan 13, 2019 at 6:03 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> Thanks!
>
> Sent from my iPhone
>
> > On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
> >
> > For the upgrade it is probably best when you change the version number in
> > [1] and then work through the build issues.
> >
> > I would expect a number of dependency changes since the client classpath
> > now has the Hadoop transitive dependencies shaded away [2].
> >
> > Regarding the development process, see [3].
> >
> > Thomas
> >
> > [1]
> >
> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
> > [2] https://hadoop.apache.org/docs/r3.0.0/
> > [3] http://apex.apache.org/contributing.html
> >
> >
> >
> >> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com>
> wrote:
> >>
> >> at first blush, I would think, that the YARN node tagging and native
> Docker
> >> support would be particularly intriguing...the GPU node support would
> be a
> >> good implementation trick to be able to put operators that do deep
> learning
> >> tasks...
> >>
> >> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <
> pramod.immaneni@gmail.com
> >>>
> >> wrote:
> >>
> >>> I agree we should just try changing the dependency first. I am in favor
> >> of
> >>> upgrading. It will be good to explore if and how we can leverage some
> of
> >>> the new features in 3.x.
> >>>
> >>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org>
> wrote:
> >>>>
> >>>> Hi Aaron,
> >>>>
> >>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
> >>>> recompiling and running unit test. Also, before you put significant
> >>> effort
> >>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
> >>>> everyone agrees and vote passes, please open JIRA for tracking the
> >>> changes.
> >>>>
> >>>> Thank you,
> >>>>
> >>>> Vlad
> >>>>
> >>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
> >> wrote:
> >>>>>
> >>>>> As requested by Thomas, I am hoping we can kick off a discussion
> >> about
> >>> a
> >>>>> short-term goal of adding support for Hadoop 3.X, though, if I am not
> >>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but will
> >>>>> confirm, in which case, the ask is to support 3.1+ specifically.
> >>>>>
> >>>>> I program in Scala for the most part, but have not problem switching
> >> to
> >>>>> pure Java as needed. That being said, I am very unfamiliar with the
> >>> Apex
> >>>>> code base and also unsure of the scope and level of effort required
> >> to
> >>>> get
> >>>>> this done. I look to the group for a few pointers on where to begin
> >> as
> >>>>> well as code style and such...I don't think I will need a lot of
> >>>>> hand-holding, just a good shove in the right direction...
> >>>>>
> >>>>> Thanks again for the discussion and support on this...for background
> >>> here
> >>>>> is my personal and professional motivation for pressing forward:
> >>>>>
> >>>>> - Doing research work for clients as well as internal R&D for
> >>> streaming
> >>>>> data processing
> >>>>> - very specific requirement to support the ability to alter
> >>>>> topologies programatically and manually on the fly, hence the
> >>>> interest in
> >>>>> Apex, which is the only platform I know of that has this support
> >>>> built
> >>>>> in...I don't want to re-invent that wheel unless absolutely
> >>>> necessary
> >>>>> - General interest in supporting the open-source community, both
> >> as a
> >>>>> company and individually
> >>>>>
> >>>>> --
> >>>>>
> >>>>> M. Aaron Bossert
> >>>>> (202) 594-1703
> >>>>> Punch Cyber Analytics Group
> >>>>
> >>>> --
> >>> Thanks,
> >>> Pramod
> >>> http://ts.la/pramod3443
> >>>
> >>
> >>
> >> --
> >>
> >> M. Aaron Bossert
> >> (202) 594-1703
> >> Punch Cyber Analytics Group
> >>
>
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
Thanks!
Sent from my iPhone
> On Jan 13, 2019, at 18:00, Thomas Weise <th...@apache.org> wrote:
>
> For the upgrade it is probably best when you change the version number in
> [1] and then work through the build issues.
>
> I would expect a number of dependency changes since the client classpath
> now has the Hadoop transitive dependencies shaded away [2].
>
> Regarding the development process, see [3].
>
> Thomas
>
> [1]
> https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
> [2] https://hadoop.apache.org/docs/r3.0.0/
> [3] http://apex.apache.org/contributing.html
>
>
>
>> On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> wrote:
>>
>> at first blush, I would think, that the YARN node tagging and native Docker
>> support would be particularly intriguing...the GPU node support would be a
>> good implementation trick to be able to put operators that do deep learning
>> tasks...
>>
>> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <pramod.immaneni@gmail.com
>>>
>> wrote:
>>
>>> I agree we should just try changing the dependency first. I am in favor
>> of
>>> upgrading. It will be good to explore if and how we can leverage some of
>>> the new features in 3.x.
>>>
>>>> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org> wrote:
>>>>
>>>> Hi Aaron,
>>>>
>>>> I’d recommend to start with upgrading Hadoop dependency to 3.x,
>>>> recompiling and running unit test. Also, before you put significant
>>> effort
>>>> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
>>>> everyone agrees and vote passes, please open JIRA for tracking the
>>> changes.
>>>>
>>>> Thank you,
>>>>
>>>> Vlad
>>>>
>>>>> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
>> wrote:
>>>>>
>>>>> As requested by Thomas, I am hoping we can kick off a discussion
>> about
>>> a
>>>>> short-term goal of adding support for Hadoop 3.X, though, if I am not
>>>>> mistaken, there was a pretty large change from 3.0 to 3.1...but will
>>>>> confirm, in which case, the ask is to support 3.1+ specifically.
>>>>>
>>>>> I program in Scala for the most part, but have not problem switching
>> to
>>>>> pure Java as needed. That being said, I am very unfamiliar with the
>>> Apex
>>>>> code base and also unsure of the scope and level of effort required
>> to
>>>> get
>>>>> this done. I look to the group for a few pointers on where to begin
>> as
>>>>> well as code style and such...I don't think I will need a lot of
>>>>> hand-holding, just a good shove in the right direction...
>>>>>
>>>>> Thanks again for the discussion and support on this...for background
>>> here
>>>>> is my personal and professional motivation for pressing forward:
>>>>>
>>>>> - Doing research work for clients as well as internal R&D for
>>> streaming
>>>>> data processing
>>>>> - very specific requirement to support the ability to alter
>>>>> topologies programatically and manually on the fly, hence the
>>>> interest in
>>>>> Apex, which is the only platform I know of that has this support
>>>> built
>>>>> in...I don't want to re-invent that wheel unless absolutely
>>>> necessary
>>>>> - General interest in supporting the open-source community, both
>> as a
>>>>> company and individually
>>>>>
>>>>> --
>>>>>
>>>>> M. Aaron Bossert
>>>>> (202) 594-1703
>>>>> Punch Cyber Analytics Group
>>>>
>>>> --
>>> Thanks,
>>> Pramod
>>> http://ts.la/pramod3443
>>>
>>
>>
>> --
>>
>> M. Aaron Bossert
>> (202) 594-1703
>> Punch Cyber Analytics Group
>>
Re: support Hadoop 3.X
Posted by Thomas Weise <th...@apache.org>.
For the upgrade it is probably best when you change the version number in
[1] and then work through the build issues.
I would expect a number of dependency changes since the client classpath
now has the Hadoop transitive dependencies shaded away [2].
Regarding the development process, see [3].
Thomas
[1]
https://github.com/apache/apex-core/blob/1f5425dc5ac1d3452a3b37db0e4c61485af9c6e1/pom.xml#L109
[2] https://hadoop.apache.org/docs/r3.0.0/
[3] http://apex.apache.org/contributing.html
On Fri, Jan 11, 2019 at 5:14 PM Aaron Bossert <aa...@punchcyber.com> wrote:
> at first blush, I would think, that the YARN node tagging and native Docker
> support would be particularly intriguing...the GPU node support would be a
> good implementation trick to be able to put operators that do deep learning
> tasks...
>
> On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <pramod.immaneni@gmail.com
> >
> wrote:
>
> > I agree we should just try changing the dependency first. I am in favor
> of
> > upgrading. It will be good to explore if and how we can leverage some of
> > the new features in 3.x.
> >
> > On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org> wrote:
> >
> > > Hi Aaron,
> > >
> > > I’d recommend to start with upgrading Hadoop dependency to 3.x,
> > > recompiling and running unit test. Also, before you put significant
> > effort
> > > into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
> > > everyone agrees and vote passes, please open JIRA for tracking the
> > changes.
> > >
> > > Thank you,
> > >
> > > Vlad
> > >
> > > > On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com>
> wrote:
> > > >
> > > > As requested by Thomas, I am hoping we can kick off a discussion
> about
> > a
> > > > short-term goal of adding support for Hadoop 3.X, though, if I am not
> > > > mistaken, there was a pretty large change from 3.0 to 3.1...but will
> > > > confirm, in which case, the ask is to support 3.1+ specifically.
> > > >
> > > > I program in Scala for the most part, but have not problem switching
> to
> > > > pure Java as needed. That being said, I am very unfamiliar with the
> > Apex
> > > > code base and also unsure of the scope and level of effort required
> to
> > > get
> > > > this done. I look to the group for a few pointers on where to begin
> as
> > > > well as code style and such...I don't think I will need a lot of
> > > > hand-holding, just a good shove in the right direction...
> > > >
> > > > Thanks again for the discussion and support on this...for background
> > here
> > > > is my personal and professional motivation for pressing forward:
> > > >
> > > > - Doing research work for clients as well as internal R&D for
> > streaming
> > > > data processing
> > > > - very specific requirement to support the ability to alter
> > > > topologies programatically and manually on the fly, hence the
> > > interest in
> > > > Apex, which is the only platform I know of that has this support
> > > built
> > > > in...I don't want to re-invent that wheel unless absolutely
> > > necessary
> > > > - General interest in supporting the open-source community, both
> as a
> > > > company and individually
> > > >
> > > > --
> > > >
> > > > M. Aaron Bossert
> > > > (202) 594-1703
> > > > Punch Cyber Analytics Group
> > >
> > > --
> > Thanks,
> > Pramod
> > http://ts.la/pramod3443
> >
>
>
> --
>
> M. Aaron Bossert
> (202) 594-1703
> Punch Cyber Analytics Group
>
Re: support Hadoop 3.X
Posted by Aaron Bossert <aa...@punchcyber.com>.
at first blush, I would think, that the YARN node tagging and native Docker
support would be particularly intriguing...the GPU node support would be a
good implementation trick to be able to put operators that do deep learning
tasks...
On Fri, Jan 11, 2019 at 2:42 PM Pramod Immaneni <pr...@gmail.com>
wrote:
> I agree we should just try changing the dependency first. I am in favor of
> upgrading. It will be good to explore if and how we can leverage some of
> the new features in 3.x.
>
> On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org> wrote:
>
> > Hi Aaron,
> >
> > I’d recommend to start with upgrading Hadoop dependency to 3.x,
> > recompiling and running unit test. Also, before you put significant
> effort
> > into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
> > everyone agrees and vote passes, please open JIRA for tracking the
> changes.
> >
> > Thank you,
> >
> > Vlad
> >
> > > On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> wrote:
> > >
> > > As requested by Thomas, I am hoping we can kick off a discussion about
> a
> > > short-term goal of adding support for Hadoop 3.X, though, if I am not
> > > mistaken, there was a pretty large change from 3.0 to 3.1...but will
> > > confirm, in which case, the ask is to support 3.1+ specifically.
> > >
> > > I program in Scala for the most part, but have not problem switching to
> > > pure Java as needed. That being said, I am very unfamiliar with the
> Apex
> > > code base and also unsure of the scope and level of effort required to
> > get
> > > this done. I look to the group for a few pointers on where to begin as
> > > well as code style and such...I don't think I will need a lot of
> > > hand-holding, just a good shove in the right direction...
> > >
> > > Thanks again for the discussion and support on this...for background
> here
> > > is my personal and professional motivation for pressing forward:
> > >
> > > - Doing research work for clients as well as internal R&D for
> streaming
> > > data processing
> > > - very specific requirement to support the ability to alter
> > > topologies programatically and manually on the fly, hence the
> > interest in
> > > Apex, which is the only platform I know of that has this support
> > built
> > > in...I don't want to re-invent that wheel unless absolutely
> > necessary
> > > - General interest in supporting the open-source community, both as a
> > > company and individually
> > >
> > > --
> > >
> > > M. Aaron Bossert
> > > (202) 594-1703
> > > Punch Cyber Analytics Group
> >
> > --
> Thanks,
> Pramod
> http://ts.la/pramod3443
>
--
M. Aaron Bossert
(202) 594-1703
Punch Cyber Analytics Group
Re: support Hadoop 3.X
Posted by Pramod Immaneni <pr...@gmail.com>.
I agree we should just try changing the dependency first. I am in favor of
upgrading. It will be good to explore if and how we can leverage some of
the new features in 3.x.
On Fri, Jan 11, 2019 at 11:38 AM Vlad Rozov <vr...@apache.org> wrote:
> Hi Aaron,
>
> I’d recommend to start with upgrading Hadoop dependency to 3.x,
> recompiling and running unit test. Also, before you put significant effort
> into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once
> everyone agrees and vote passes, please open JIRA for tracking the changes.
>
> Thank you,
>
> Vlad
>
> > On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> wrote:
> >
> > As requested by Thomas, I am hoping we can kick off a discussion about a
> > short-term goal of adding support for Hadoop 3.X, though, if I am not
> > mistaken, there was a pretty large change from 3.0 to 3.1...but will
> > confirm, in which case, the ask is to support 3.1+ specifically.
> >
> > I program in Scala for the most part, but have not problem switching to
> > pure Java as needed. That being said, I am very unfamiliar with the Apex
> > code base and also unsure of the scope and level of effort required to
> get
> > this done. I look to the group for a few pointers on where to begin as
> > well as code style and such...I don't think I will need a lot of
> > hand-holding, just a good shove in the right direction...
> >
> > Thanks again for the discussion and support on this...for background here
> > is my personal and professional motivation for pressing forward:
> >
> > - Doing research work for clients as well as internal R&D for streaming
> > data processing
> > - very specific requirement to support the ability to alter
> > topologies programatically and manually on the fly, hence the
> interest in
> > Apex, which is the only platform I know of that has this support
> built
> > in...I don't want to re-invent that wheel unless absolutely
> necessary
> > - General interest in supporting the open-source community, both as a
> > company and individually
> >
> > --
> >
> > M. Aaron Bossert
> > (202) 594-1703
> > Punch Cyber Analytics Group
>
> --
Thanks,
Pramod
http://ts.la/pramod3443
Re: support Hadoop 3.X
Posted by Vlad Rozov <vr...@apache.org>.
Hi Aaron,
I’d recommend to start with upgrading Hadoop dependency to 3.x, recompiling and running unit test. Also, before you put significant effort into Hadoop 3.x upgrade, let’s discuss and vote on dev@apex. Once everyone agrees and vote passes, please open JIRA for tracking the changes.
Thank you,
Vlad
> On Jan 10, 2019, at 07:58, Aaron Bossert <aa...@punchcyber.com> wrote:
>
> As requested by Thomas, I am hoping we can kick off a discussion about a
> short-term goal of adding support for Hadoop 3.X, though, if I am not
> mistaken, there was a pretty large change from 3.0 to 3.1...but will
> confirm, in which case, the ask is to support 3.1+ specifically.
>
> I program in Scala for the most part, but have not problem switching to
> pure Java as needed. That being said, I am very unfamiliar with the Apex
> code base and also unsure of the scope and level of effort required to get
> this done. I look to the group for a few pointers on where to begin as
> well as code style and such...I don't think I will need a lot of
> hand-holding, just a good shove in the right direction...
>
> Thanks again for the discussion and support on this...for background here
> is my personal and professional motivation for pressing forward:
>
> - Doing research work for clients as well as internal R&D for streaming
> data processing
> - very specific requirement to support the ability to alter
> topologies programatically and manually on the fly, hence the interest in
> Apex, which is the only platform I know of that has this support built
> in...I don't want to re-invent that wheel unless absolutely necessary
> - General interest in supporting the open-source community, both as a
> company and individually
>
> --
>
> M. Aaron Bossert
> (202) 594-1703
> Punch Cyber Analytics Group