You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nemo.apache.org by wo...@apache.org on 2018/02/28 06:59:12 UTC
[incubator-nemo] branch master created (now 6bdf2db)
This is an automated email from the ASF dual-hosted git repository.
wonook pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-nemo.git.
at 6bdf2db [Nemo-794] Add SparkSQL Examples and Integration Tests. (#795)
This branch includes the following new commits:
new e3fa003 first commit
new b65cbbe set up a new Maven project (#3)
new 764a836 Add Build Status Badge to README.md (#5)
new bc46471 Initial Working Code (#26)
new 4925e43 Compiler Interfaces (#33)
new bc0d706 Update README.md
new 57295e1 Upgrade Beam to 0.4.0 (#40)
new d59ab93 Code Cleanup and Clarification (#42)
new ad9ac79 [Nemo-39] Execution Plan Builder (#43)
new a69fa85 23 stylechecker (#49)
new d7780b5 12 sink node (#51)
new 2cd1f66 Change terminology for edge types (#55)
new 7e963ad [Nemo-29] Configurable optimizer (#53)
new 3e48787 [Nemo-21] Refactor Attributes Class (#74)
new 1655c14 [Nemo-76] Revised IR Definition (#77)
new 26985fd [Nemo-79] Refactor Execution Plan (#99)
new 157ed24 [Minor] Internal change in execution plan builder (#101)
new 7c2cd95 [Minor] Import mockito for unit testing (#102)
new 2329fc0 [Nemo-28] Nemo Backend Execution Plan (#73)
new acd5a1e [Nemo-106] Add Edge type to Attributes (#107)
new f3bcd49 [Nemo-80] Convert logical execution plan to physical plan (#105)
new 0122918 [Minor] Make Runtime Exceptions 'final' (#112)
new 3182d06 [Nemo-108] Refactor IR and runtime attributes (#109)
new 63e6960 [Nemo-78] reimplement broadcast (#110)
new 1c476c4 [Nemo-82] Introduce messaging components (#114)
new 8ede705 [Nemo-104] change getReaders interface with desired split number (#115)
new 697b03c [Nemo-122] State Machine (#123)
new 3907958 add messages to exceptions (#126)
new d8c08b1 [Nemo-57] Example program ALS (#117)
new c549137 [Nemo-84] Introduce channels (#124)
new 5b0cbc9 [Nemo-81] Introduce Scheduler (#120)
new dee152f [Minor] Log the Generated Physical Plan (#127)
new 4348907 [Nemo-125] Implement Local File Manager (#128)
new 2fd1d71 [Nemo-129] Add interfaces for parent/children vertex information in DAG (#130)
new fcbfc4e [Nemo-#52] Receive optimization policies as a parameter (#133)
new 57ad9a3 [Nemo-131] Pass for required attributes (#136)
new 978d969 max instead of sum (#137)
new 4d3f646 [Minor] Divide stages where edges involve sideinputs
new a48f853 [Nemo-#90] Integrate Components for Single-Machine End-to-End Execution (#134)
new 89a5b67 update jenkind badge to use https, as updated (#143)
new 6cd8cc8 [Nemo-135] Use Beam 0.6.0 (#142)
new ec54765 [Nemo-30] Use Tang to Parse User Arguments (#145)
new efb8962 [Nemo-132] Refactor DAG (#151)
new beb6146 [Nemo-57] Add Multinomial Logistic Regression workload (#147)
new c9b7e4e [Nemo-150] Composite Transforms in loops of BEAM applications. (#155)
new 13d0ee1 [Nemo-156] get rid of side input filter (#157)
new c8e3cf3 Introduce states (#158)
new eebda07 [Nemo-140] DAG Visualization (#166)
new 2966aa2 [Nemo-167] Instructions/Scripts for DAG visualization (#169)
new 54aa6ab [Nemo-168] Clean up logging (#170)
new 42488d5 [Nemo-88, 93] Batch Round-Robin Scheduler, Execution State Manager (#165)
new bbdd884 [Nemo-91, 71] Task InputReader, OutputWriter (#174)
new a05dde9 [Minor] DAG - match vertices by ID (#175)
new 7f93fc0 [Nemo-87] Organize Block States and Introduce Block Manager (#179)
new ed4d49c [Nemo-182] Define Control Messages (#183)
new f92339f [Nemo-121] introduce loop (#161)
new b1bae02 serializable reader (#185)
new c9143dd [Nemo-189,190] Javadocs for compiler
new ac7d956 [Nemo-83, 85, 184] Integrate Runtime Components (#188)
new b1e510d [Nemo-194] Configuration File for Runtime (#195)
new 73aaea5 [Nemo-160] Tests for the compiler side and optimization passes. (#198)
new 134dd8b [Nemo-186] BlockManager Communication with Messages (#196)
new d707596 [Nemo-18] Preserve coder information in IREdge (#203)
new efb91a3 [Minor] Visualizer improvements #204
new 7b05b5a [Nemo-44] Import REEF (#209)
new 22ce338 [Nemo-207] Remove PhysicalPlan from Executors (#215)
new 6cfcfa5 [Nemo-197] Use Coder for Serialization/Deserialization (#216)
new 79e251c remove (#221)
new 3409977 [Nemo-176] Implement LoopFusionPass (#214)
new a9d9786 [Nemo-227] Separate Integration Tests from Unit Tests (#228)
new c11cedd [Nemo-217] Schedule Stages in a Recursive Manner (#231)
new 0ca7a93 [Nemo-240] Print dag after each optimization (#241)
new 4082ffe [Nemo-212] Run Nemo on YARN (#243)
new 14ea2b3 [Nemo-144] make default policy work (#244)
new 1086ae4 [Nemo-233] Introduce Container Manager (#247)
new 73a00f3 [Nemo-238] Visualize Job State with Physical DAG (#239)
new 8c6c1c6 [Nemo-199] Block Transfer using Wake (#229)
new 865750b [Nemo-234] Add Scheduler Test (#251)
new 8cff838 [Nemo-237] Introduce "deploy" directory (#253)
new dae849d Change Default DAG directory (#255)
new b2f5179 [Nemo-210] Update Beam Version to 2.0.0 (#260)
new c6bb580 [Nemo-232] Fix No Free Port Bug (#263)
new e094584 [Minor] Rename ResourceType to ContainerType (#261)
new 568baf1 suppress warnings for iterative algorithms (#259)
new 03fd12c [Nemo-248] Run 1GB MapReduce on YARN (#269)
new 9f6f9c5 [Nemo-267] Fix No Free Port Bug 2 (#272)
new e52d851 [Nemo-177] Common Subexpression Elimination Pass (#256)
new 998f528 [Nemo-270] Fix BatchSchedulerTest (#278)
new a673c03 [Nemo-220] Serialization Codecs For Beam Applications (#282)
new 3d72f5e [Nemo-277] Run 12GB MapReduce (#281)
new 8796385 [Nemo-286] Report Executor's Failure Stack Trace to Driver (#288)
new 6fe643f [Nemo-22] DAG integrity check (#284)
new ea41fa8 [Minor] add timeout to integration tests (#289)
new 52bd145 [Minor] Fixes for tests: update build status badge and temporarily disable some ITs for speed (#293)
new 80b6732 [Nemo-208] Check for Job Termination in a Cleaner Way (#287)
new 8c94f27 [Nemo-149] More beam applications (#297)
new 3da1e9b [Nemo-302] Rename block to partition (#303)
new 903d6f8 [Nemo-242] LoopVertex Visualization (#307)
new 6d532e5 [Nemo-163] Fault Tolerance (#305)
new b56ba3c [Nemo-296] Resolve parallelism equality check failure (#308)
new 9657d7b [Minor] Partition transfer without protobuf (#309)
new 5067bc4 [Nemo-250] Fetch multiple partitions in parallel (#314)
new 8047ac7 [Nemo-69] Implement file channel in Runtime (#304)
new e23ab1b [Nemo-310] DynOpt-AddDataSkewPass (#317)
new 1839e28 [Nemo-312] DynOpt-Fix Nemo Master for Dynamic Optimization (#318)
new 7e69b67 [Nemo-321] Fix FaultToleranceTest
new 4f2a3c5 get rid of runtime attributes (#324)
new 88650e5 Our CI server now supports HTTPS! (#330)
new 0adfb4f [Nemo-329] Use NESTED context for encoding/decoding using BeamCoder (#331)
new d2680af [Nemo-319] Integration test doesn't run with File attribute as default (#332)
new ad4fce8 [Nemo-328] Don't split stages when using File (#333)
new 00ffd16 [Nemo-301] Divide a Task's Output Partitions into Smaller Blocks (#327)
new f88b592 [Nemo-336] Make getPartition return before obtaining PartitionLocation from master (#341)
new 673bd70 [Nemo-334] Interface change in executor for data push (#340)
new 623a404 DAG AttributeCheck: SideInput must be non-push, one-to-one (#342)
new d5e5575 [Nemo-335] Introduce producer - consumer structure to task group execution (#343)
new c9ee74b [Nemo-337] PendingTaskGroupPriorityQueue (#344)
new 78073e2 [Nemo-325] Get rid of logical DAG and its components (#345)
new 9a67689 [Minor] Introduce ReplyFutureMap (#346)
new f66f989 [Nemo-313] Dynamic Optimization - Metric collection and transferring them to Master (#349)
new ab8f433 [Nemo-338, 348] Refactor Partition States (#351)
new d455a9f [Nemo-350] Remove Ambiguous External Vertex AttributeMap (#352)
new 01cb34f [Nemo-339] onRequestPartitionLocation can wait until the partition becomes available (#357)
new efceb3e Remove Storage from ContainerType (#371)
new 2522e35 Add missing comments for interfaces (#368)
new 6c5a8ba [Nemo-359] Enable "Push" in BatchScheduler (#360)
new 2053305 Fix missing argument for log (#376)
new d1b9820 [Nemo-372] Clean up Store Types (#373)
new 857cf71 Fix intermittent DataTransferTest failure (#375)
new 4788af0 [Minor] Make FilePartition reuse FileOutputStream (#377)
new 9a4b687 fix pado pass (#379)
new a3e2bb4 Minor fix in PendingTGPQ (#380)
new 6d90d82 [Nemo-326] Hash-aware block partitioning (#358)
new 13d496d [Nemo-315] dynamic optimization: resubmit DAG to runtime (#363)
new 88d97cf Remove unnecessary extra scheduling attempt (#381)
new 119b08e Rename TestUtils && Fix BatchSchedulerTest mocking problem (#382)
new 3e7e8b7 [Nemo-205] Richer resource configuration schema (#385)
new bcec7b3 [Nemo-180] Implement DistributedStorageStore (#386)
new 0713561 Add protobuf installation (#389)
new be4ed25 Prevent data collection with push data transfer (#393)
new 295606f [Minor] Config Directory (#394)
new e06ebb2 [Nemo-392] SLF4J Logger (#398)
new 9118c13 Add additional transition (#399)
new 8854c2c [Nemo-362] dyn opt support hash aware block partitioning (#391)
new 40dea16 [Nemo-387] Threading on PartitionStore & PartitionTransferPeer (#400)
new a7de9c9 [Nemo-403] Abstract the metadata access layer from *FilePartition (#406)
new 0541d44 [Nemo-397] Fix PendingTaskGroupPriorityQueue Concurrency Issue (#413)
new 1d77172 [Minor] Fix MapReduce problem. (#424)
new e292acf commit (#425)
new 76a805e [Nemo-235] Clean up Task State Management (#415)
new ff860ab [Nemo-418] Collect Executor Metric in Master (#422)
new d0b7cee [Nemo-404] Introduce metadata server model (#426)
new eafd73a [Nemo-417] Executor metric format (#435)
new 53043f5 [Minor] Adds FileArea and HashRange (#441)
new 5606b8c [Nemo-355, 300] Support I-file write (#434)
new 42339a2 [Nemo-426] Stage Partitioning Pass (#439)
new d420d82 [Nemo-423] DynOpt: Use PubSubEventHandler for handling events between compiler and runtime (#429)
new 17394ca [Nemo-173] Implement Data Transfer Protocol (#442)
new d1f1e04 Add a command line parameter: JVM heap slack (#453)
new 2a7245e [Minor] cleanups following #429 (#454)
new dfcacd7 [Minor] Improve job state logging (#457)
new 6b26acf [Nemo-450] Channel setup for the data plane (#452)
new da5c81a [Nemo-460] Setting up transport context (#461)
new ad30940 remove (#468)
new 10d997f [Nemo-390] DynOpt: finalize data skew handling policy (#455)
new 846672e [Nemo-448] Use REEF 0.17.0-SNAPSHOT (#467)
new cb5f910 [Nemo-437] Introduce schedule group and pass (#459)
new 7fbad20 [Nemo-465] End-to-end stream-based API (#466)
new fe75cc8 clean up stream apis with code convention (#474)
new 2be1402 [Nemo-265] Close All Requested Executors (#486)
new 3755432 [Nemo-487] ScheduleGroup in PhysicalPlan (#488)
new 39dce75 [Nemo-440] Refactor data plane (#490)
new 23b300b [Nemo-475, 444] Unload PartitionStore (#476)
new aa61c9d [Nemo - 419, 443] Instrument execution metrics (#456)
new 0da6de2 [Minor] Fix container failrue message (#499)
new 34962b7 [Nemo-438] Fix BatchScheduler to Schedule According to ScheduleGroup (#496)
new 5468b32 Fix partition state control in task group failed case (#500)
new 7c07a11 [Nemo-481] Remove Priority Dependencies when Container Types are Different (#502)
new 2203396 [Nemo-470] Enable custom policy of passes (#482)
new ad3bef2 [Nemo-430, 431] refactor partition & block state managing in master side (#483)
new b8686d7 [Nemo-473] Refactor attributes (#497)
new f2dc452 [Nemo-510] Make NcsMessageEnvironment Flexible (#512)
new b57163b [Nemo-505] Beam on vortex guide (#503)
new 3a6e296 Support listener removal in MessageEnvironment (#514)
new 419e523 [Nemo-523] Get rid of IREdge type and unify it into DataCommunicationPattern ExecutionProperty (#524)
new 5236021 [Nemo-479] Reorganize passes (#516)
new 06808d5 Fix ContainerManager (#526)
new 5efa112 [Nemo-527] Introduce MetricManager (#531)
new 489c16a [Minor] Modify IR visualizer to work for refactored attributes (#534)
new f154e9c [Nemo-522] Remove CompletableFutures in the write / read process (#533)
new 6dabf39 [Nemo-536] Pick out io thread from partitionstore (#537)
new 595d6f2 [Minor] Better job completion message
new 3c1b473 [Nemo-540] Make scatter-gather to create a single file (#542)
new e7a4507 [Nemo-493] Detach partitioning from writing (#539)
new 5384d95 [Nemo-480] Remove Stage Dependency Information from PendingTaskGroupQueue
new 4583ad1 [Nemo-515] Update policies to use rules and JSON (#530)
new d0d1c4b [Nemo-546] Remove current I-File write implementation & multi source task situations (#549)
new fa1aede [Nemo-548] Rename old name to Nemo (#551)
new d476911 [Nemo-544] Enable operator task to emit data multiple times (#552)
new 26a67e1 Fix void coder bug (key of an element can be null) (#554)
new 7a182fe s3a (#555)
new c3f5f93 [Nemo-274, 416] Beam hdfs write (#553)
new c5d24ce [Minor] Improve IR Visualizer (#556)
new a31c2e8 minor - don't sent metric message (#559)
new 47290cf commit (#557)
new 1059f81 change (#560)
new 631e801 add (#561)
new d79e2c0 [Minor] Hand over disk configurations to executors (#562)
new 63fc24e [Minor] DAG: Add getVertexById (#532)
new f375e13 [Minor] Add Container Fail Type #566
new e86ccfc [Minor] Catch and throw exception during NCS message handling #565
new 7547881 [Minor] Add Partition Transitions for Output Failures #569
new 1bbbf4b [Minor] Collect metrics from master only #568
new 96ce93a [Nemo-547] Implement I-File Merger Operator (#564)
new b3a3da5 [Minor] Fix Synchronization Issues in PendingTGPQ
new 272ea7d [Minor] Add Beam's .temp Files to .gitignore
new e0f88e2 [Minor] Fix source parallelism of one to one comm (#573)
new c134c4f [Minor] Synchronize Local File Metadata List (#570)
new 6708109 [Minor] Delete stale file (#574)
new aef6d7c [Minor] policy builder unit test (#580)
new a5cbc44 [Nemo-538, 577] Reverse Topologically Schedule a ScheduleGroup (#579)
new 329ba23 [Nemo-584] Prevent Repetitive Scheduling of a TaskGroup during Faults (#585)
new 460400c [Nemo-582] Remove element (#583)
new c6138d7 [Nemo-594] NemoDriver to Access Runtime through RuntimeMaster #595
new 1b17b32 [Nemo-596] Separate Beam pipeline compilation from the driver (#597)
new f87e48b [Nemo-588] Implement MemoryStore which stores data in serialized form (#589)
new d8cfc20 [Minor] Throw IOException in Coder (Instead of RuntimeException) #599
new 2c59085 [Minor] Disable Concurrent Write in FilePartition (#600)
new 632ec6b [Nemo-598] Update 0.17.0-SNAPSHOT reef jars in our repository
new 5ae126c [Nemo-206] Replace NCS with a proper RPC framework (#590)
new 36017a6 [Nemo-593] Refactor Scheduler Components (#603)
new 8a84860 [Nemo-602] Add Fault Tolerance Test (#604)
new d1a728b [Nemo-24] pom.xml per Sub-Package (#616)
new 8f14bd3 [Minor] Avoid NullPointerException in getting ExecutionProperty.Value
new 1532d15 move partitioner (#619)
new 06ccacb [Minor] Fix a bug that prevented the visualizer from handling Tasks (#621)
new 2950b4b [Minor] Fix dag .json directory into one (#624)
new 990e6a6 [Nemo-606] Support put / get serialized block in each PartitionStore (#625)
new 769a722 [Nemo-383] Review default execution properties and passes (#627)
new 9f6ea93 [Nemo-298] Remove Used Blocks from Memory (#632)
new a60581a [Minor] Generate dag directory for each ITs (#634)
new 0eadfb2 [Nemo-311] Avoid copy overhead on serialization from Stream to byte array (#638)
new 46f1278 [Nemo-636] Implement extra encoding & decoding task for Sailfish (#640)
new 8f4b5ea [Nemo-458] Add additional appender to log4j.properties (#644)
new 20099d4 [Nemo-610] Decouple Nemo Job and REEF-related components (#646)
new 57d3d29 [Minor] Don't close filesystem at finishBundle
new 2c6d0bf [Nemo-653] Add license information to project root (#654)
new f31e00b [Nemo-650] Add license information to script files (#652)
new 7ba53f3 [Nemo-119] Javadoc for methods (#656)
new 7abe7c0 [Minor] Switch Block with Partition (#663)
new ab58fed [Nemo-367] Clean up beam elements (#664)
new 04fec3c [Nemo-494] Replace Hash Range with Key Range (#665)
new be98bd1 [Minor] Improve transform names for visualization (#666)
new a03a931 [Minor] Fix configuration for bundled Maven repository (#667)
new 0476866 [Nemo-660] Remove TODOs that reference closed issues (#668)
new 6fc2c50 [Nemo-647] Add license information to pom files (#670)
new 821e4be [Minor] Fix Used Data Handling #672
new 27f1c0b [Nemo-671] Change ${version} to ${project.version} in pom.xmls (#673)
new c037073 [Nemo-651] Fix Broken Links in Javadoc (#669)
new c827b0d [Nemo-630] Add separate CoderManager class (#674)
new 80f8082 [Nemo-119] Re-enable method javadocs requirement for checkstyle plugin (#675)
new dd9a552 split project pom file into multiple pom files (#648)
new 9747b9c Fix dependencies (#679)
new 83cc562 [Nemo-642] Add onyx web ui package (#655)
new 092d3fe [Nemo-159] Forward Remote Master Logs to Client (#680)
new 9902411 [Nemo-236] Handle Control Events in a Separate Thread Pool (#661)
new 49b2951 switch to ncs message environment (#690)
new 49ec8e2 [Minor] Fix resolving local repository (#694)
new 774adca [Nemo-692] Support Java9 (#693)
new 734d513 [Nemo-633, 658, 285] Refactor SchedulingPolicy (#697)
new c361f90 [Nemo-695] Make WriteFiles to be executed properly (#699)
new 7d8de9a [Nemo-698] Remove Beam dependency from project root (#700)
new fb4a49c [Nemo-702] Fix BeamCoder to catch CoderException and throw IOException instead (#703)
new 50b1f32 [Minor] Stop encoding thread on channel write exception (#706)
new aa29917 [Nemo-701] Refactor Beam-related components into a separate package (#705)
new fbc982e [Nemo-469] Check test output correctness (#704)
new 13f132c Make control messages to be GCed (#708)
new b2cd9af [Nemo-709] Wait for Executor Launch before Scheduling TaskGroups (#710)
new 7bd400e Separate log4j.properties (#716)
new 573c24d [Nemo-715] Remote Block Metadata in File (#718)
new 6104c04 Use Iterator (instead of Iterable) in InputReading and feeding Transform (#720)
new 234ca14 [Nemo-721] Make CompressionProperty for configurable compressor options (#722)
new 53334d1 [Nemo-411] Nemo to support Spark with an MR example (SparkPi) (#712)
new 83efadb [Nemo-727] Update parallelism pass (#728)
new 2def128 [Minor] Remove Sailfish Codec Tasks
new 9c74219 [Nemo-723] Make TaskGroup executor thread to do decoding (#726)
new a74a6ad [Minor] Inquiry block location to master before checking local store (#730)
new f0d8233 [Minor] Fix readme and example script
new 9edbe8c [Nemo-581] Integration test with multi-parallelism (#732)
new 8d8d7d6 [Minor] BroadcastTransform to CreateViewTransform (#735)
new 69ccce0 [Minor] Increase timeout for MapReduceITCase and BroadcastITCase (#736)
new 4cf8a3e [Nemo-707] Make TaskGroup executor thread to do encoding (#731)
new fb46153 [Nemo-734] Implement CompressionPass (#741)
new 99e52c9 [Nemo-724] Execution Time Metric Clean up (#725)
new 6a18946 [Nemo-743] Split Source in Executor Instead of Driver
new 760c7bb [Nemo-748,749] Fix ALS and related passes/tests (#750)
new d47c940 [Nemo-742] Make PhysicalPlan scalable (#746)
new 9149bdf [Minor] Serialize TaskGroup Scheduling Message in Separate Executor Service (#752)
new 9a923f2 [Minor] Fix stage partitioning algorithm (#751)
new de3af82 [Nemo-747] Remove Task Group Class (#755)
new ae0194d [Minor] Fix race condition on TaskGroupState/TaskState (#757)
new 570701c [Minor] Fix Broadcast Edge Annotation (#754)
new 7752f95 [Nemo-711] Spark word count (#745)
new 8ee0d16 [Nemo-686] Travis (#760)
new b4fd4ad [Nemo-279] Make Visualizer Scalable (#759)
new 88a921a [Minor] Travis support + reconfigure timeouts for IT cases (#761)
new e92db1e [Minor] Rename onyx to coral (#763)
new c60800e [Minor] Get Readables in Master (#764)
new 5e3c29f [Nemo-762] Support multiple byte streams for one block transfer (#765)
new 8380bda [Nemo-567] Implement compression (#758)
new 9513399 [Minor] Update dependencies on README.md (#768)
new 97a7155 [Nemo-769] Fix `getPartitions` to work with compression (#775)
new 580d936 [Nemo-756] Distribution of Spark Source (#777)
new 3bacd57 [Nemo-773] Revise data skew policy (#774)
new 2081204 [Nemo-529] Subscription of Runtime Event Handlers (#780)
new 670442a [Minor] Rename Coral to Nemo (#783)
new e8878c0 [Minor] Do not instantiate separate PubSubEventHandler (#785)
new a398daf [Minor] Fix Travis behavior (#784)
new 981b205 [Minor] Add Task Level Metric & Script (#778)
new a15480d [Minor] Add metric data on data input (#786)
new e7fd59a [Minor] Add missing license headers (#788)
new 67614db [Minor] Add Scala support + Scala Spark Examples (#779)
new 1de2023 [Nemo-781] Support SparkSQL (#791)
new 4f980b2 [Nemo-792] Fix loop bugs (#793)
new 6bdf2db [Nemo-794] Add SparkSQL Examples and Integration Tests. (#795)
The 325 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
--
To stop receiving notification emails like this one, please contact
wonook@apache.org.