You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nemo.apache.org by jo...@apache.org on 2019/01/21 06:18:25 UTC
[incubator-nemo] branch 326-ctpass created (now 31b65a6)
This is an automated email from the ASF dual-hosted git repository.
johnyangk pushed a change to branch 326-ctpass
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git.
at 31b65a6 chkpt
This branch includes the following new commits:
new e3fa003 first commit
new b65cbbe set up a new Maven project (#3)
new 764a836 Add Build Status Badge to README.md (#5)
new bc46471 Initial Working Code (#26)
new 4925e43 Compiler Interfaces (#33)
new bc0d706 Update README.md
new 57295e1 Upgrade Beam to 0.4.0 (#40)
new d59ab93 Code Cleanup and Clarification (#42)
new ad9ac79 [Nemo-39] Execution Plan Builder (#43)
new a69fa85 23 stylechecker (#49)
new d7780b5 12 sink node (#51)
new 2cd1f66 Change terminology for edge types (#55)
new 7e963ad [Nemo-29] Configurable optimizer (#53)
new 3e48787 [Nemo-21] Refactor Attributes Class (#74)
new 1655c14 [Nemo-76] Revised IR Definition (#77)
new 26985fd [Nemo-79] Refactor Execution Plan (#99)
new 157ed24 [Minor] Internal change in execution plan builder (#101)
new 7c2cd95 [Minor] Import mockito for unit testing (#102)
new 2329fc0 [Nemo-28] Nemo Backend Execution Plan (#73)
new acd5a1e [Nemo-106] Add Edge type to Attributes (#107)
new f3bcd49 [Nemo-80] Convert logical execution plan to physical plan (#105)
new 0122918 [Minor] Make Runtime Exceptions 'final' (#112)
new 3182d06 [Nemo-108] Refactor IR and runtime attributes (#109)
new 63e6960 [Nemo-78] reimplement broadcast (#110)
new 1c476c4 [Nemo-82] Introduce messaging components (#114)
new 8ede705 [Nemo-104] change getReaders interface with desired split number (#115)
new 697b03c [Nemo-122] State Machine (#123)
new 3907958 add messages to exceptions (#126)
new d8c08b1 [Nemo-57] Example program ALS (#117)
new c549137 [Nemo-84] Introduce channels (#124)
new 5b0cbc9 [Nemo-81] Introduce Scheduler (#120)
new dee152f [Minor] Log the Generated Physical Plan (#127)
new 4348907 [Nemo-125] Implement Local File Manager (#128)
new 2fd1d71 [Nemo-129] Add interfaces for parent/children vertex information in DAG (#130)
new fcbfc4e [Nemo-#52] Receive optimization policies as a parameter (#133)
new 57ad9a3 [Nemo-131] Pass for required attributes (#136)
new 978d969 max instead of sum (#137)
new 4d3f646 [Minor] Divide stages where edges involve sideinputs
new a48f853 [Nemo-#90] Integrate Components for Single-Machine End-to-End Execution (#134)
new 89a5b67 update jenkind badge to use https, as updated (#143)
new 6cd8cc8 [Nemo-135] Use Beam 0.6.0 (#142)
new ec54765 [Nemo-30] Use Tang to Parse User Arguments (#145)
new efb8962 [Nemo-132] Refactor DAG (#151)
new beb6146 [Nemo-57] Add Multinomial Logistic Regression workload (#147)
new c9b7e4e [Nemo-150] Composite Transforms in loops of BEAM applications. (#155)
new 13d0ee1 [Nemo-156] get rid of side input filter (#157)
new c8e3cf3 Introduce states (#158)
new eebda07 [Nemo-140] DAG Visualization (#166)
new 2966aa2 [Nemo-167] Instructions/Scripts for DAG visualization (#169)
new 54aa6ab [Nemo-168] Clean up logging (#170)
new 42488d5 [Nemo-88, 93] Batch Round-Robin Scheduler, Execution State Manager (#165)
new bbdd884 [Nemo-91, 71] Task InputReader, OutputWriter (#174)
new a05dde9 [Minor] DAG - match vertices by ID (#175)
new 7f93fc0 [Nemo-87] Organize Block States and Introduce Block Manager (#179)
new ed4d49c [Nemo-182] Define Control Messages (#183)
new f92339f [Nemo-121] introduce loop (#161)
new b1bae02 serializable reader (#185)
new c9143dd [Nemo-189,190] Javadocs for compiler
new ac7d956 [Nemo-83, 85, 184] Integrate Runtime Components (#188)
new b1e510d [Nemo-194] Configuration File for Runtime (#195)
new 73aaea5 [Nemo-160] Tests for the compiler side and optimization passes. (#198)
new 134dd8b [Nemo-186] BlockManager Communication with Messages (#196)
new d707596 [Nemo-18] Preserve coder information in IREdge (#203)
new efb91a3 [Minor] Visualizer improvements #204
new 7b05b5a [Nemo-44] Import REEF (#209)
new 22ce338 [Nemo-207] Remove PhysicalPlan from Executors (#215)
new 6cfcfa5 [Nemo-197] Use Coder for Serialization/Deserialization (#216)
new 79e251c remove (#221)
new 3409977 [Nemo-176] Implement LoopFusionPass (#214)
new a9d9786 [Nemo-227] Separate Integration Tests from Unit Tests (#228)
new c11cedd [Nemo-217] Schedule Stages in a Recursive Manner (#231)
new 0ca7a93 [Nemo-240] Print dag after each optimization (#241)
new 4082ffe [Nemo-212] Run Nemo on YARN (#243)
new 14ea2b3 [Nemo-144] make default policy work (#244)
new 1086ae4 [Nemo-233] Introduce Container Manager (#247)
new 73a00f3 [Nemo-238] Visualize Job State with Physical DAG (#239)
new 8c6c1c6 [Nemo-199] Block Transfer using Wake (#229)
new 865750b [Nemo-234] Add Scheduler Test (#251)
new 8cff838 [Nemo-237] Introduce "deploy" directory (#253)
new dae849d Change Default DAG directory (#255)
new b2f5179 [Nemo-210] Update Beam Version to 2.0.0 (#260)
new c6bb580 [Nemo-232] Fix No Free Port Bug (#263)
new e094584 [Minor] Rename ResourceType to ContainerType (#261)
new 568baf1 suppress warnings for iterative algorithms (#259)
new 03fd12c [Nemo-248] Run 1GB MapReduce on YARN (#269)
new 9f6f9c5 [Nemo-267] Fix No Free Port Bug 2 (#272)
new e52d851 [Nemo-177] Common Subexpression Elimination Pass (#256)
new 998f528 [Nemo-270] Fix BatchSchedulerTest (#278)
new a673c03 [Nemo-220] Serialization Codecs For Beam Applications (#282)
new 3d72f5e [Nemo-277] Run 12GB MapReduce (#281)
new 8796385 [Nemo-286] Report Executor's Failure Stack Trace to Driver (#288)
new 6fe643f [Nemo-22] DAG integrity check (#284)
new ea41fa8 [Minor] add timeout to integration tests (#289)
new 52bd145 [Minor] Fixes for tests: update build status badge and temporarily disable some ITs for speed (#293)
new 80b6732 [Nemo-208] Check for Job Termination in a Cleaner Way (#287)
new 8c94f27 [Nemo-149] More beam applications (#297)
new 3da1e9b [Nemo-302] Rename block to partition (#303)
new 903d6f8 [Nemo-242] LoopVertex Visualization (#307)
new 6d532e5 [Nemo-163] Fault Tolerance (#305)
new b56ba3c [Nemo-296] Resolve parallelism equality check failure (#308)
new 9657d7b [Minor] Partition transfer without protobuf (#309)
new 5067bc4 [Nemo-250] Fetch multiple partitions in parallel (#314)
new 8047ac7 [Nemo-69] Implement file channel in Runtime (#304)
new e23ab1b [Nemo-310] DynOpt-AddDataSkewPass (#317)
new 1839e28 [Nemo-312] DynOpt-Fix Nemo Master for Dynamic Optimization (#318)
new 7e69b67 [Nemo-321] Fix FaultToleranceTest
new 4f2a3c5 get rid of runtime attributes (#324)
new 88650e5 Our CI server now supports HTTPS! (#330)
new 0adfb4f [Nemo-329] Use NESTED context for encoding/decoding using BeamCoder (#331)
new d2680af [Nemo-319] Integration test doesn't run with File attribute as default (#332)
new ad4fce8 [Nemo-328] Don't split stages when using File (#333)
new 00ffd16 [Nemo-301] Divide a Task's Output Partitions into Smaller Blocks (#327)
new f88b5925 [Nemo-336] Make getPartition return before obtaining PartitionLocation from master (#341)
new 673bd70 [Nemo-334] Interface change in executor for data push (#340)
new 623a404 DAG AttributeCheck: SideInput must be non-push, one-to-one (#342)
new d5e5575 [Nemo-335] Introduce producer - consumer structure to task group execution (#343)
new c9ee74b [Nemo-337] PendingTaskGroupPriorityQueue (#344)
new 78073e2 [Nemo-325] Get rid of logical DAG and its components (#345)
new 9a67689 [Minor] Introduce ReplyFutureMap (#346)
new f66f989 [Nemo-313] Dynamic Optimization - Metric collection and transferring them to Master (#349)
new ab8f433 [Nemo-338, 348] Refactor Partition States (#351)
new d455a9f [Nemo-350] Remove Ambiguous External Vertex AttributeMap (#352)
new 01cb34f [Nemo-339] onRequestPartitionLocation can wait until the partition becomes available (#357)
new efceb3e Remove Storage from ContainerType (#371)
new 2522e35 Add missing comments for interfaces (#368)
new 6c5a8ba [Nemo-359] Enable "Push" in BatchScheduler (#360)
new 2053305 Fix missing argument for log (#376)
new d1b9820 [Nemo-372] Clean up Store Types (#373)
new 857cf71 Fix intermittent DataTransferTest failure (#375)
new 4788af0 [Minor] Make FilePartition reuse FileOutputStream (#377)
new 9a4b687 fix pado pass (#379)
new a3e2bb4 Minor fix in PendingTGPQ (#380)
new 6d90d82 [Nemo-326] Hash-aware block partitioning (#358)
new 13d496d [Nemo-315] dynamic optimization: resubmit DAG to runtime (#363)
new 88d97cf Remove unnecessary extra scheduling attempt (#381)
new 119b08e Rename TestUtils && Fix BatchSchedulerTest mocking problem (#382)
new 3e7e8b7 [Nemo-205] Richer resource configuration schema (#385)
new bcec7b3 [Nemo-180] Implement DistributedStorageStore (#386)
new 0713561 Add protobuf installation (#389)
new be4ed25 Prevent data collection with push data transfer (#393)
new 295606f [Minor] Config Directory (#394)
new e06ebb2 [Nemo-392] SLF4J Logger (#398)
new 9118c13 Add additional transition (#399)
new 8854c2c [Nemo-362] dyn opt support hash aware block partitioning (#391)
new 40dea16 [Nemo-387] Threading on PartitionStore & PartitionTransferPeer (#400)
new a7de9c9 [Nemo-403] Abstract the metadata access layer from *FilePartition (#406)
new 0541d44 [Nemo-397] Fix PendingTaskGroupPriorityQueue Concurrency Issue (#413)
new 1d77172 [Minor] Fix MapReduce problem. (#424)
new e292acf commit (#425)
new 76a805e [Nemo-235] Clean up Task State Management (#415)
new ff860ab [Nemo-418] Collect Executor Metric in Master (#422)
new d0b7cee [Nemo-404] Introduce metadata server model (#426)
new eafd73a [Nemo-417] Executor metric format (#435)
new 53043f5 [Minor] Adds FileArea and HashRange (#441)
new 5606b8c [Nemo-355, 300] Support I-file write (#434)
new 42339a2 [Nemo-426] Stage Partitioning Pass (#439)
new d420d82 [Nemo-423] DynOpt: Use PubSubEventHandler for handling events between compiler and runtime (#429)
new 17394ca [Nemo-173] Implement Data Transfer Protocol (#442)
new d1f1e04 Add a command line parameter: JVM heap slack (#453)
new 2a7245e [Minor] cleanups following #429 (#454)
new dfcacd7 [Minor] Improve job state logging (#457)
new 6b26acf [Nemo-450] Channel setup for the data plane (#452)
new da5c81a [Nemo-460] Setting up transport context (#461)
new ad30940 remove (#468)
new 10d997f [Nemo-390] DynOpt: finalize data skew handling policy (#455)
new 846672e [Nemo-448] Use REEF 0.17.0-SNAPSHOT (#467)
new cb5f910 [Nemo-437] Introduce schedule group and pass (#459)
new 7fbad20 [Nemo-465] End-to-end stream-based API (#466)
new fe75cc8 clean up stream apis with code convention (#474)
new 2be1402 [Nemo-265] Close All Requested Executors (#486)
new 3755432 [Nemo-487] ScheduleGroup in PhysicalPlan (#488)
new 39dce75 [Nemo-440] Refactor data plane (#490)
new 23b300b [Nemo-475, 444] Unload PartitionStore (#476)
new aa61c9d [Nemo - 419, 443] Instrument execution metrics (#456)
new 0da6de2 [Minor] Fix container failrue message (#499)
new 34962b7 [Nemo-438] Fix BatchScheduler to Schedule According to ScheduleGroup (#496)
new 5468b32 Fix partition state control in task group failed case (#500)
new 7c07a11 [Nemo-481] Remove Priority Dependencies when Container Types are Different (#502)
new 2203396 [Nemo-470] Enable custom policy of passes (#482)
new ad3bef2 [Nemo-430, 431] refactor partition & block state managing in master side (#483)
new b8686d7 [Nemo-473] Refactor attributes (#497)
new f2dc452 [Nemo-510] Make NcsMessageEnvironment Flexible (#512)
new b57163b [Nemo-505] Beam on vortex guide (#503)
new 3a6e296 Support listener removal in MessageEnvironment (#514)
new 419e523 [Nemo-523] Get rid of IREdge type and unify it into DataCommunicationPattern ExecutionProperty (#524)
new 5236021 [Nemo-479] Reorganize passes (#516)
new 06808d5 Fix ContainerManager (#526)
new 5efa112 [Nemo-527] Introduce MetricManager (#531)
new 489c16a [Minor] Modify IR visualizer to work for refactored attributes (#534)
new f154e9c [Nemo-522] Remove CompletableFutures in the write / read process (#533)
new 6dabf39 [Nemo-536] Pick out io thread from partitionstore (#537)
new 595d6f2 [Minor] Better job completion message
new 3c1b473 [Nemo-540] Make scatter-gather to create a single file (#542)
new e7a4507 [Nemo-493] Detach partitioning from writing (#539)
new 5384d95 [Nemo-480] Remove Stage Dependency Information from PendingTaskGroupQueue
new 4583ad1 [Nemo-515] Update policies to use rules and JSON (#530)
new d0d1c4b [Nemo-546] Remove current I-File write implementation & multi source task situations (#549)
new fa1aede [Nemo-548] Rename old name to Nemo (#551)
new d476911 [Nemo-544] Enable operator task to emit data multiple times (#552)
new 26a67e1 Fix void coder bug (key of an element can be null) (#554)
new 7a182fe s3a (#555)
new c3f5f93 [Nemo-274, 416] Beam hdfs write (#553)
new c5d24ce [Minor] Improve IR Visualizer (#556)
new a31c2e8 minor - don't sent metric message (#559)
new 47290cf commit (#557)
new 1059f81 change (#560)
new 631e801 add (#561)
new d79e2c0 [Minor] Hand over disk configurations to executors (#562)
new 63fc24e [Minor] DAG: Add getVertexById (#532)
new f375e13 [Minor] Add Container Fail Type #566
new e86ccfc [Minor] Catch and throw exception during NCS message handling #565
new 7547881 [Minor] Add Partition Transitions for Output Failures #569
new 1bbbf4b [Minor] Collect metrics from master only #568
new 96ce93a [Nemo-547] Implement I-File Merger Operator (#564)
new b3a3da5 [Minor] Fix Synchronization Issues in PendingTGPQ
new 272ea7d [Minor] Add Beam's .temp Files to .gitignore
new e0f88e2 [Minor] Fix source parallelism of one to one comm (#573)
new c134c4f [Minor] Synchronize Local File Metadata List (#570)
new 6708109 [Minor] Delete stale file (#574)
new aef6d7c [Minor] policy builder unit test (#580)
new a5cbc44 [Nemo-538, 577] Reverse Topologically Schedule a ScheduleGroup (#579)
new 329ba23 [Nemo-584] Prevent Repetitive Scheduling of a TaskGroup during Faults (#585)
new 460400c [Nemo-582] Remove element (#583)
new c6138d7 [Nemo-594] NemoDriver to Access Runtime through RuntimeMaster #595
new 1b17b32 [Nemo-596] Separate Beam pipeline compilation from the driver (#597)
new f87e48b [Nemo-588] Implement MemoryStore which stores data in serialized form (#589)
new d8cfc20 [Minor] Throw IOException in Coder (Instead of RuntimeException) #599
new 2c59085 [Minor] Disable Concurrent Write in FilePartition (#600)
new 632ec6b [Nemo-598] Update 0.17.0-SNAPSHOT reef jars in our repository
new 5ae126c [Nemo-206] Replace NCS with a proper RPC framework (#590)
new 36017a6 [Nemo-593] Refactor Scheduler Components (#603)
new 8a84860 [Nemo-602] Add Fault Tolerance Test (#604)
new d1a728b [Nemo-24] pom.xml per Sub-Package (#616)
new 8f14bd3 [Minor] Avoid NullPointerException in getting ExecutionProperty.Value
new 1532d15 move partitioner (#619)
new 06ccacb [Minor] Fix a bug that prevented the visualizer from handling Tasks (#621)
new 2950b4b [Minor] Fix dag .json directory into one (#624)
new 990e6a6 [Nemo-606] Support put / get serialized block in each PartitionStore (#625)
new 769a722 [Nemo-383] Review default execution properties and passes (#627)
new 9f6ea93 [Nemo-298] Remove Used Blocks from Memory (#632)
new a60581a [Minor] Generate dag directory for each ITs (#634)
new 0eadfb2 [Nemo-311] Avoid copy overhead on serialization from Stream to byte array (#638)
new 46f1278 [Nemo-636] Implement extra encoding & decoding task for Sailfish (#640)
new 8f4b5ea [Nemo-458] Add additional appender to log4j.properties (#644)
new 20099d4 [Nemo-610] Decouple Nemo Job and REEF-related components (#646)
new 57d3d29 [Minor] Don't close filesystem at finishBundle
new 2c6d0bf [Nemo-653] Add license information to project root (#654)
new f31e00b [Nemo-650] Add license information to script files (#652)
new 7ba53f3 [Nemo-119] Javadoc for methods (#656)
new 7abe7c0 [Minor] Switch Block with Partition (#663)
new ab58fed [Nemo-367] Clean up beam elements (#664)
new 04fec3c [Nemo-494] Replace Hash Range with Key Range (#665)
new be98bd1 [Minor] Improve transform names for visualization (#666)
new a03a931 [Minor] Fix configuration for bundled Maven repository (#667)
new 0476866 [Nemo-660] Remove TODOs that reference closed issues (#668)
new 6fc2c50 [Nemo-647] Add license information to pom files (#670)
new 821e4be [Minor] Fix Used Data Handling #672
new 27f1c0b [Nemo-671] Change ${version} to ${project.version} in pom.xmls (#673)
new c037073 [Nemo-651] Fix Broken Links in Javadoc (#669)
new c827b0d [Nemo-630] Add separate CoderManager class (#674)
new 80f8082 [Nemo-119] Re-enable method javadocs requirement for checkstyle plugin (#675)
new dd9a552 split project pom file into multiple pom files (#648)
new 9747b9c Fix dependencies (#679)
new 83cc562 [Nemo-642] Add onyx web ui package (#655)
new 092d3fe [Nemo-159] Forward Remote Master Logs to Client (#680)
new 9902411 [Nemo-236] Handle Control Events in a Separate Thread Pool (#661)
new 49b2951 switch to ncs message environment (#690)
new 49ec8e2 [Minor] Fix resolving local repository (#694)
new 774adca [Nemo-692] Support Java9 (#693)
new 734d513 [Nemo-633, 658, 285] Refactor SchedulingPolicy (#697)
new c361f90 [Nemo-695] Make WriteFiles to be executed properly (#699)
new 7d8de9a [Nemo-698] Remove Beam dependency from project root (#700)
new fb4a49c [Nemo-702] Fix BeamCoder to catch CoderException and throw IOException instead (#703)
new 50b1f32 [Minor] Stop encoding thread on channel write exception (#706)
new aa29917 [Nemo-701] Refactor Beam-related components into a separate package (#705)
new fbc982e [Nemo-469] Check test output correctness (#704)
new 13f132c Make control messages to be GCed (#708)
new b2cd9af [Nemo-709] Wait for Executor Launch before Scheduling TaskGroups (#710)
new 7bd400e Separate log4j.properties (#716)
new 573c24d [Nemo-715] Remote Block Metadata in File (#718)
new 6104c04 Use Iterator (instead of Iterable) in InputReading and feeding Transform (#720)
new 234ca14 [Nemo-721] Make CompressionProperty for configurable compressor options (#722)
new 53334d1 [Nemo-411] Nemo to support Spark with an MR example (SparkPi) (#712)
new 83efadb [Nemo-727] Update parallelism pass (#728)
new 2def128 [Minor] Remove Sailfish Codec Tasks
new 9c74219 [Nemo-723] Make TaskGroup executor thread to do decoding (#726)
new a74a6ad [Minor] Inquiry block location to master before checking local store (#730)
new f0d8233 [Minor] Fix readme and example script
new 9edbe8c [Nemo-581] Integration test with multi-parallelism (#732)
new 8d8d7d6 [Minor] BroadcastTransform to CreateViewTransform (#735)
new 69ccce0 [Minor] Increase timeout for MapReduceITCase and BroadcastITCase (#736)
new 4cf8a3e [Nemo-707] Make TaskGroup executor thread to do encoding (#731)
new fb46153 [Nemo-734] Implement CompressionPass (#741)
new 99e52c9 [Nemo-724] Execution Time Metric Clean up (#725)
new 6a18946 [Nemo-743] Split Source in Executor Instead of Driver
new 760c7bb [Nemo-748,749] Fix ALS and related passes/tests (#750)
new d47c940 [Nemo-742] Make PhysicalPlan scalable (#746)
new 9149bdf [Minor] Serialize TaskGroup Scheduling Message in Separate Executor Service (#752)
new 9a923f2 [Minor] Fix stage partitioning algorithm (#751)
new de3af82 [Nemo-747] Remove Task Group Class (#755)
new ae0194d [Minor] Fix race condition on TaskGroupState/TaskState (#757)
new 570701c [Minor] Fix Broadcast Edge Annotation (#754)
new 7752f95 [Nemo-711] Spark word count (#745)
new 8ee0d16 [Nemo-686] Travis (#760)
new b4fd4ad [Nemo-279] Make Visualizer Scalable (#759)
new 88a921a [Minor] Travis support + reconfigure timeouts for IT cases (#761)
new e92db1e [Minor] Rename onyx to coral (#763)
new c60800e [Minor] Get Readables in Master (#764)
new 5e3c29f [Nemo-762] Support multiple byte streams for one block transfer (#765)
new 8380bda [Nemo-567] Implement compression (#758)
new 9513399 [Minor] Update dependencies on README.md (#768)
new 97a7155 [Nemo-769] Fix `getPartitions` to work with compression (#775)
new 580d936 [Nemo-756] Distribution of Spark Source (#777)
new 3bacd57 [Nemo-773] Revise data skew policy (#774)
new 2081204 [Nemo-529] Subscription of Runtime Event Handlers (#780)
new 670442a [Minor] Rename Coral to Nemo (#783)
new e8878c0 [Minor] Do not instantiate separate PubSubEventHandler (#785)
new a398daf [Minor] Fix Travis behavior (#784)
new 981b205 [Minor] Add Task Level Metric & Script (#778)
new a15480d [Minor] Add metric data on data input (#786)
new e7fd59a [Minor] Add missing license headers (#788)
new 67614db [Minor] Add Scala support + Scala Spark Examples (#779)
new 1de2023 [Nemo-781] Support SparkSQL (#791)
new 4f980b2 [Nemo-792] Fix loop bugs (#793)
new 6bdf2db [Nemo-794] Add SparkSQL Examples and Integration Tests. (#795)
new a7721ac [NEMO-787] Pack Block Location Request Messaging (#796)
new a887423 [Minor] Correct default path for executor configuration (#797)
new ee830d5 [Nemo-772] Reuse Duplicate data (#790)
new 2ca28cc [NEMO-29] Update GitHub Main README.md (#3)
new 0745ac5 receive element in BlockStore & refactor partitioner interface
new a065ea0 [NEMO-28] Javadoc and Travis (#4)
new 80878f0 Merge remote-tracking branch 'upstream-inc/master' into 27-ElementWiseWrite
new d65227d debug dataskew
new 46af499 disable concurrent write in Blocks
new 7d8b9c3 [NEMO-26] Implement SourceLocationAwareSchedulingPolicy (#1)
new f5e8545 resolve conflict
new f5103a3 handle block in BlockManagerWorker instead of BlockStore
new d30e25e [NEMO-34] SonarCloud (#7)
new 4b7244f Merge branch 'master' into 27-ElementWiseWrite
new da7aa8e Merge branch 'master' into 27-ElementWiseWrite
new 61864b8 [NEMO-40] Solve SonarCloud issues for Frontend and Examples (#9)
new a88aaf2 Merge branch 'master' into 27-ElementWiseWrite
new bd05b5a [NEMO-36] Make executor do not care about capcacity (#8)
new 0ea7451 Merge branch 'master' into 27-ElementWiseWrite
new 46a2dcc [NEMO-39] SonarCloud Bugs and Vulnerabilities for RuntimeExecutor
new ca3f347 [NEMO-43] Sonar cloud script (#13)
new de065b8 [NEMO-31] Community Standards (#10)
new f03ed05 [NEMO-42] Make SchedulerRunner reactive, Ensure reverse-topological ordering in scheduling (#11)
new 033d814 [NEMO-38] Fix bugs and vulnerabilities for RuntimeMaster and RuntimeCommon (#14)
new cf041f6 [NEMO-7] Intra-TaskGroup pipelining (#2)
new 3f22abd Merge branch '27-ElementWiseWrite' of github.com:snuspl/incubator-nemo into 27-ElementWiseWrite
new 3cf3403 merge pipelining
new fef3019 minor fix
new de77ec1 move code
new b3d00dc address comment
new 3a7177a tests pass
new f36d2e2 [NEMO-53] Make SchedulingPolicy Stackable (#15)
new aabc7c7 merge
new c68f306 Merge remote-tracking branch 'apache/master' into 27-ElementWiseWrite
new fabed6a address comments
new 27e2852 Merge pull request #5 from snuspl/27-ElementWiseWrite
new ec0b846 Merge branch 'master' into 66-tests-scheduler
new 2105af6 move basic policies
new fbc39ba Merge pull request #17 from johnyangk/66-tests-scheduler
new 9196c20 [NEMO-47] Use primitive types for AlternatingLeastSquare (#19)
new 2958d84 [NEMO-76] Rename TaskGroup to Task (#20)
new 70b15a9 [NEMO-46] Make the operations on ExecutorRegistry atomic (#21)
new 4da2fc8 [NEMO-45] Distributed Nemo-Spark (#18)
new 94761f4 [NEMO-67] Fix Communication Pattern for Beam CoGroupByKey transform (#23)
new 0d9a6ec [NEMO-68] Restrict the number of parallel connections between executors (#22)
new 9941c1c [NEMO-79] Clean up the legacy Task (#24)
new 1fbe99c [NEMO-49] Replace failed executor with a new executor (#27)
new d418580 [NEMO-78] Rename PhysicalStage to Stage (#26)
new a387c4c [NEMO-95] Rename ExecutableTask to Task (#29)
new 4d3ef80 [NEMO-12] Frontend support for Scala Spark (#28)
new ed205fe [NEMO-63] On-demand dynamic optimization metric aggregation (#30)
new 8df268f [NEMO-97] Refactor TaskExecutor and fix a sideinput bug (#31)
new 7ac7478 [NEMO-101] Make Coder as an execution property (#32)
new c294d34 [NEMO-105] Move /tests/common into /common/test (#34)
new 02eb7c0 [NEMO-35] Controlling logging levels while running tests (#35)
new 997abf2 [NEMO-114] Update copyright year for 2018 (#37)
new e92926c [NEMO-74] Rename RoundRobinSchedulingPolicy (#40)
new f912822 [Nemo 108] SonarCloud Bugs and Vulnerabilities for Nemo Runtime Master (#41)
new 3362d3b [NEMO-48] Do not merge broadcasted data into a single partition (#38)
new a8849bf [NEMO-43] Handle InterruptedException properly for Nemo Client (#43)
new e224c5d [NEMO-112] Handle InterruptedException properly (#39)
new 31ed262 [NEMO-107] SonarCloud Bugs for Nemo Executor (#46)
new a266d40 [NEMO-83] Move /tests/runtime into /runtime/tests (#47)
new ff8a120 [NEMO-94] Clean up legacy TODOs (#36)
new 9b0b133 [NEMO-113] Update README (#44)
new 2c93e7b [NEMO-41] SonarCloud Bugs and Vulnerabilities for NemoCommon (#42)
new 8b79671 [NEMO-111] Remove ExecutionProperty.Key (#33)
new 88b7933 [NEMO-87] Remove unused BlockStates (#49)
new 39482ae [NEMO-103] Implement RPC between Client and Driver (#45)
new 5dd93ac [NEMO-72] Instance-based Encoder/Decoder interface (#48)
new 7ef822b [NEMO-32] Support as byte[] encoding & decoding (#50)
new add2b55 [NEMO-102] Stage Partitioning by PhysicalPlanGenerator (#51)
new 3ac2759 [NEMO-122] Manage Task/Stage/Job states in one place (#52)
new aa3bf8b [NEMO-123] Replace PendingTaskCollection with pointers to ScheduleGroups (#53)
new 9e6124c [NEMO-127] Modify DefaultInterTaskDataStorePass not to overwrite property #55
new 9e58c32 [NEMO-126] Split ScheduleGroup by pull StageEdges by PhysicalPlanGenerator (#54)
new 3493fba [NEMO-16] Implement collection of data from executor to client (#56)
new 6db5b07 [NEMO-133] Remove run_external_app.sh (#58)
new 209369c [NEMO-73,75] SchedulingPolicy as Vertex-level Execution Property (#57)
new b621434 [NEMO-50] Carefully retry tasks in the scheduler (#59)
new b88accb [NEMO-61] Fix lost execution metric collection problem (#60)
new c7bdd27 [NEMO-124] Support DoFn#output(tag, output) (#63)
new 5305e6b [NEMO-104] Move /tests/client into /client/test (#64)
new df44252 [NEMO-54] Handle remote data fetch failures due to executor removal (#67)
new c3bd780 [NEMO-125] Fix data loss bug caused by SailfishSchedulingPolicy (#68)
new 9fe914c [NEMO-20] RESTful APIs to Access Job State and Metric (#61)
new 100224c [NEMO-113] Update README (#70)
new 64a0912 [NEMO-71] Add NodeNamesAssignmentPass and Example Application (#62)
new 7439962 [NEMO-59] Skewed data-aware executor allocation (#72)
new 6804b98 [NEMO-55] Handle NCS Master-to-Executor RPC failures (#71)
new 0af0383 [NEMO-141] Make vertices receiving push edge not comply executor slot (#74)
new 9c83146 [NEMO-161] Make all injectable constructors private (#77)
new c7e39fb [NEMO-163] Update README.md to use Hadoop 2.7.2 (#78)
new ad5386f [NEMO-2, 106, 117] Remove the /tests directory (#69)
new 3e9b400 [NEMO-164] Reduce Overhead of FreeSlotSchedulingConstraint (#79)
new 1a39efa [NEMO-165] Bug when a task reads from multiple parent tasks
new 0da3f85 [NEMO-62] Support Multiple DAG Submission in a Single User Program (#73)
new 2597079 [NEMO-166] Do not attach NodeNamesProperty to Source Stages (#82)
new e93facc [NEMO-167] Rename Passes and Execution Properties (#83)
new 6d35ddd [NEMO-171] Fix confusing ITCase input/output file names (#84)
new 16c5e73 [NEMO-151] Add OutputWriters for addtional tagged outputs (#87)
new 65ad522 [NEMO-81] Support the Beam 'Partition' transform (#90)
new 8f4e7f5 [NEMO-51] Intermediate data location aware scheduling (#86)
new a749280 [NEMO-138] Rule-based policy (#88)
new 9333cf3 [NEMO-172] Implement one partition per one element partitioner (#85)
new 661497e [NEMO-175] Remove 'taskInfo', add Scheduler#completeHeldTask (#89)
new 435a274 [NEMO-176] Improve sequential read from disk #92
new 0c42189 [NEMO-152] Support Multiple DAGs Execution in a Single User Program (#94)
new 0ec68b6 [NEMO-160] Handle Beam VoidCoder properly (#93)
new 54dc2ca [NEMO-177] Fix BytesDecoder to handle empty stream properly (#96)
new 04d88fe [NEMO-136] Rename SchedulerRunner to TaskDispatcher (#97)
new 1122f32 [NEMO-118] Implement Skipping (De)Serialization by ExecutionProperty (#98)
new d5df47c [NEMO-168] Fine-grained Execution Property Check (#101)
new 1b9bfa0 [NEMO-3] Bump up the Beam version to 2.5.0 (#91)
new d59373f [NEMO-64] Fix map stage hang under DataSkewPolicy (#99)
new f49f58f [NEMO-184] Upgrade to Beam 2.6.0 (#105)
new 4e09f9a [NEMO-183] DAG-centric translation from Beam pipeline to IR DAG (#104)
new ac233d5 [NEMO-142] Add .editorconfig and formatter.xml (#106)
new 98dbe68 [NEMO-178] Zero-delay task cloning (#107)
new 1fc9791 [NEMO-192] Remove IRDAG from PhysicalPlan (#108)
new 46519a6 [NEMO-180] Java Annotation for Prerequisite Execution Properties (#102)
new f1283ff [NEMO-89] Clean up IRVertex#getClone() (#109)
new faecf75 [NEMO-197] Close JIRA issues upon closing PRs (#110)
new e6c3616 [NEMO-139, 6] Logic in the scheduler for appending jobs, Support RDD caching (#111)
new 25bea60 [NEMO-179] Delayed Task Cloning (#112)
new 954d92f [NEMO-15] Run Spark ALS in distributed mode (#113)
new 3dcff47 [NEMO-96] Modularize DataSkewPolicy to use MetricVertex and BarrierVertex (#115)
new 36513ad [NEMO-211] Move Java package and Maven artifact from edu.snu.nemo to org.apache.nemo
new 51b2d16 [NEMO-212] Move package directories to org/apache/nemo (#117)
new 84b4a0c [NEMO-203] Beam SQL Aggregation fails to match RowCoder in Combine transform (#114)
new c16cb8b [NEMO-215] Add beam-runners-core-java package to nemo-beam-frontend (#119)
new 2ec8f28 [NEMO-214] Use Jackson for building JSON, fix malformed JSON issue on Windows (#118)
new 223caf2 [NEMO-217] Immediately emits elements in FlattenTransform (#120)
new fa75565 [NEMO-218] Throws exception while processing non-keyed events in CreateViewTransform (#121)
new 18b61c1 [NEMO-213] Use Beam's DoFnRunners to execute DoFn (#122)
new c4170bf [NEMO-129] Support Beam's WindowedWordCount example (#123)
new 210dfe1 [NEMO-225] Drop REEF JARs from source tree (#127)
new d2b016e [NEMO-224] Simple StreamingScheduler (#126)
new 13cb991 [NEMO-128] Support Beam UnboundedSource (#125)
new 251440a [NEMO-235] Refactor TaskExecutor's data processing logic (#128)
new 1709e95 [NEMO-239] Edit Source Headers for ASF Guideline Compliance (#132)
new 503b6de [Nemo-240] Fix .travis.yml (#133)
new 5c03023 [Nemo-238] Optimize processElement in TaskExecutor (#131)
new 970751a [NEMO-233] Emit watermark at unbounded source (#130)
new 0c826c2 [NEMO-8] Implement PipeManagerMaster/Worker (#129)
new 5b12d6c [NEMO-231] Add onWatermark() method to Transform (#134)
new 728a79d [NEMO-246] Add javadoc generation to CI (#136)
new eea3f57 [NEMO-248] Validate javadoc in mvn phase (#139)
new 04d6eb4 [NEMO-230] Emit collected data when receiving watermark in GroupByKeyAndWindowTransform (#135)
new 53522c6 [NEMO-237] Refactor ParentTaskDataFetcher to emit streaming data and watermark (#138)
new 7858051 [NEMO-232] Implement InputWatermarkManager (#137)
new 6e4a317 [NEMO-254] Modify POM to publish maven artifacts (#140)
new a726a5e [NEMO-252] Fix CreatViewTransform to emit windowed materialized data (#141)
new a4dcbe3 [NEMO-241] ASCII art for Nemo job launching (#143)
new d2acb4f [NEMO-242] Parent POM.xmls for \compiler, \runtime, and \examples (#144)
new 2b3a315 [Nemo-220] Move PIPELINE variable to TranslationContext in PipelineTranslator (#145)
new f34324e [NEMO-202] Make job launcher handle empty user_args #147
new 3626ae8 [NEMO-266] Throws NoSuchElementException in Readeable.readCurrent #149
new 8759f3f [NEMO-269] Direct translation from Beam DAG to Nemo DAG (#150)
new 837dc3e [NEMO-268] Consider start/finish bundles in Transform (#152)
new 61207a5 [NEMO-260] Beam Accumulator-based Partial Aggregation (#148)
new b1dad52 [NEMO-267] Consider watermark holds in GroupByKeyAndWindowDoFnTransform (#151)
new 7171f59 JIRA: [NEMO-275: Eager Garbage Collection for GroupByKey]
new dbad69d [NEMO-272] Fix incorrect uses of Beam FinishBundle (#154)
new c5ef7ed [NEMO-279] Finish BeamUnboundedSource when emitting TIMESTAMP_MAX_VALUE (#156)
new dc3519f [NEMO-245,247] Handle watermark in OutputWriter and Implement unbounded word count example (#153)
new 8c25359 [NEMO-280] Add SCM definition to the top-level POM #157
new 3105608 [NEMO-283] Exclude DAG files from RAT (#158)
new 6dd88b4 [maven-release-plugin] prepare release nemo-project-0.1
new e885de3 [maven-release-plugin] prepare for next development iteration
new a9b0675 [maven-release-plugin] rollback release tag
new 393c3da [maven-release-plugin] prepare release nemo-pproject-0.1
new 99ea87d [maven-release-plugin] prepare for next development iteration
new cdd5317 [maven-release-plugin] rollback release tag
new 63baa75 [maven-release-plugin] prepare release nemo-project-0.1
new e9e8695 [maven-release-plugin] rollback the release of nemo-project-0.1
new 64e62b8 [maven-release-plugin] rollback release tag
new f921959 [maven-release-plugin] prepare release nemo-project-0.1
new 9d02565 [maven-release-plugin] prepare for next development iteration
new fa63ad0 [ROLLBACK DURING RELEASE] Revert version from 0.2-SNAPSHOT to 0.1-SNAPSHOT
new 7d9452d [NEMO-216,251,259] Support side inputs and windowing (#159)
new 8e249c5 [NEMO-2186] Parallelism=1 for PCollectionView (#160)
new 79c5e05 [NEMO-300] Fix starvation when handling multiple pending data fetchers #161
new 8af55d8 [NEMO-303] Use Surefire 3.0.0-M1 (#162)
new 9341b21 [NEMO-244] Organize integration tests and resources (#146)
new 4e40550 [NEMO-305] Add DISCLAIMER (#168)
new d2c5c44 [NEMO-293] OOM exception in streaming (#164)
new 6e2a4db [NEMO-294] Beam-Runner (#163)
new b417cdb [NEMO-306] Add license checkstyle (#171)
new b2a65eb [NEMO-307] Enable integration test (#173)
new c2fdd43 [NEMO-270] Test different triggers in GroupByKeyAndWindowDoFnTransformTest (#172)
new 10c4008 [NEMO-311] Add StreamingPolicy #174
new d8bdaca [NEMO-10] Handle Method Javadocs Requirements for Checkstyle Warnings (#170)
new 99d4451 [NEMO-10] Handle Method Javadocs Requirements for Checkstyle Warnings (#169)
new 48f2fcb [NEMO-315] Remove checkstyle settings for javadoc error (#176)
new 671aa55 [NEMO-316] CombinePartial/FinalTransform is not disabled in streaming mode (#177)
new f6b8db3 [NEMO-310] Exclude beam/spark examples jars from deployment (#175)
new 08ff41d [NEMO-221] Confusing error messages for a failure at the client-side initialization (#165)
new 299564e [NEMO-80] SLF4J: Failed to load class org.slf4j.impl.StaticLoggerBinder (#166)
new cbd9a31 [NEMO-25] Improve WebUI to use RESTful APIs by Nemo (#95)
new 3825443 [NEMO-319] Fix path to beam resources in examples in README (#180)
new be4c48b [NEMO-159] Nemo Web UI (#181)
new d66ec54 [NEMO-11] Generalize Equality of Int Predicates for Loops (#103)
new 1403a3d update README (#185)
new 226d37f [NEMO-323] Upgrade current working version to 0.2-SNAPSHOT
new a838ae1 [NEMO-322] Committer's Guide (#186)
new 2d0f781 [NEMO-321] Fix the data skew pass metric mismatch (#188)
new 31b65a6 chkpt
The 533 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.