You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amaterasu.apache.org by ya...@apache.org on 2017/10/28 09:05:37 UTC
[incubator-amaterasu] branch master created (now d4a05e5)
This is an automated email from the ASF dual-hosted git repository.
yaniv pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-amaterasu.git.
at d4a05e5 Merge pull request #48 from nadav-har-tzvi/feature/conda-dependencies-fixed
This branch includes the following new commits:
new d8abb79 initial commit
new c414070 added buggy file dist
new ad41b96 fixed isseue with local jar dist
new 2d78d5e added logging and decline offer
new cedcdb9 change task execution
new e90d2e1 change task execution
new edc1cbd started to build the execution engine, very early stages
new 2f9fa30 started action tracking in zookeeper
new 2d9f324 added some job schedualing logic, still very initial
new cd5233f added initial git integration + test
new c2f2880 added initial YAML parsing, should probebly rethink Job/actions design when not on painkillers
new a05a2ff deleted repo folder
new 2608d5b this is a shit version, just commiting before a messive refactore
new 4857c0e refactored the job parses and started work on the job scheduler
new 110ee57 achievement unlocked: ugly scala tail recursion
new d2258e3 added ErrorAction parsin (still not in the workflow)
new 175e85f added error actions workflow (missing tests)
new a966002 added tests got erroractions workflow
new c473542 added not working spark scalla runner
new fe156a5 added a readme
new f69df27 Merge pull request #2 from roadan/master
new 0b54b6e Merge pull request #1 from roadan/sparkactionrunner
new 7a56426 added a job launcher
new 11ec014 Merge branch 'sparkactionrunner'
new 383fcf5 commit in order to test on mesos cluster
new dab7bd2 Merge branch 'master' of https://github.com/roadan/amaterasu into roadan-master
new 5239e45 Merge branch 'roadan-master'
new c8af630 Move to the Scala interpreter
new aa58492 merge
new 38a1816 Merge branch 'spark-scala-interpreter'
new 68aaff8 added executor handling in the JobScheduler
new 66b5f97 just making it build, need to fix with Karel
new 669dfee Merge branch 'task12' As per pull request created by Yaniv.
new 45313c8 merging from the roadan repo
new 8bb3087 Merge pull request #17 from shintoio/tasks-12-7-8
new 3b38e25 changed the .gitignore to ignore nsc generated files
new e75c998 Merge branch 'master' of https://github.com/shintoio/amaterasu
new 0039c40 changed the .gitignore to ignore nsc generated files
new f0e1aa5 added handling for TASK_RUNNING
new 3df9216 added JobBootstrapper for creating JobManagers, and restoring mid execution state
new ad33dde fixed JobManager bootrapping bug
new 53c3321 can restore a job from zk, still not called from scheduler
new 5a36af4 added support for resuming a job (can't be tested until the executor is implemented)
new 7310688 added the run command to the readme
new 9ce3d6f Merge pull request #22 from shintoio/task18
new 16ac11d Pass action type and source to the executor via system property
new 84fcc77 AmaContext accessible with sc and sqlContext support
new ed29a3d AmaContext accessible with sc and sqlContext support
new 34b6fa0 rdd is persisted DF not
new 3caee42 not working, for Karel to debug
new 66ed7d8 dataframes are persisted between tasks
new 955b134 cleand up build def
new c923e0d DF and RDDs are accessible between actions
new 8306396 Merge pull request #23 from shintoio/amacontext
new 98aad1b Implement actions executor
new 3b11cf7 tests pass, job fails to find the src folder on executor
new 11c6f0b cleanup and tests fixed
new 780206b added logging to the scala runner
new acd4a47 more cleaning up
new 1c8dfe7 Fix reference to source in Maki file
new 6a16cc2 Add main class to manifest
new 5c57612 fixed issues with uneeded uris now failing on port unavailable
new d47d446 added some logic to finish faild jobs
new f2f6b7f added handling for failing mid job (tests needed
new e6baa2b now failing on missing mster, need to add env
new 7350003 failing on java.lang.NoClassDefFoundError: org/apache/mesos/protobuf/GeneratedMessage
new 846ee85 its alive! action start executes but finishes with an error for some reason
new 4769e6f added a shell script to run a job
new 057acfa Copy ama-start to target folder
new 4381d53 Move ama-start to src/main/scripts. Create SBT task copyRes to copy resource files in src/main/script over to target folder
new ef7e030 Start simeple HTTP server to serve Mesos dependency files.
new ad92826 Document how to start serving Mesos dependencies
new e2b2736 Change target to Scala 2.10.5
new 2f9d81a time traveld
new 45cc231 strange merge
new 0656f4c Rename SBT task that copies scripts to targe folder to a more intuitive copyScripts name
new 5aaa914 Remove unusued code from scheduler
new b961b36 Assign a different executor ID per task prefixed with the task ID
new 98d4d3d Merge pull request #24 from shintoio/actions-executor
new f084d03 some progress with classpath
new e6afc25 small fixes
new 4b545ba added s3 support
new 8ab32bc simple end to end workflows run successfully
new 0b67374 fixed parquet for df
new 36a13d8 Merge pull request #25 from shintoio/scala-repl
new 19eef62 environment is comming from the configuration
new 497a1dc finishing touches to jobReport
new 3f9a55b some cleanup
new 6c3422a Merge pull request #26 from shintoio/env
new d70156d runner now sends execution messages
new 1e5789e with code level report
new 53828d2 execution report is printed correctly
new 58169ca Merge pull request #27 from shintoio/execution-report
new 5c8e796 better task start up time
new 3d35f0e changed logo
new 42cba35 Merge pull request #28 from shintoio/executor-perf-improvement
new 4cb3f95 cleanup before v1 release
new 39d5cd7 Merge pull request #29 from shintoio/c1-cleanup
new 0b678d1 downloading dependencies
new 9bb4b81 jars are added to spark
new 476cc07 Merge pull request #30 from shintoio/dependencies-manager
new ced076d fixed error handling bug
new 40f4211 resulved merege issue
new dbb0131 fixed issue when no deps exist
new d393ad5 Merge pull request #31 from shintoio/fix-deps-bug
new 622b569 added a download link
new 2bc6b36 fixes to the readme
new de977d5 Initial commit for R Runner
new 7e59d7e mid work
new 2b1c009 mid work
new 5afcd8e Some refactoring for the ReplUtils
new e3bd760 more refactoring for the SparkScalaRunner, to support more generic creation, so more runners ca be integrated
new ddfe8e3 Merge pull request #32 from shintoio/feature/SparkRRunner
new 59e1318 Some more refactoring to the infrastructure
new 5bb0d49 Merge pull request #33 from shintoio/feature/SparkRRunner
new 6cd4189 Jetty Http Server added with template code
new 745ae71 small bug fixes
new 75c17ca First jetty working version added
new 21d5091 Replaced Python webserver with Jetty webserver. Parameterized webserver port, spark version and web server base directory
new d85f626 Removed dummy Httpserver config file
new 67139fc Update amaterasu.properties
new b5716b3 Merge branch 'master' into jetty_http_server
new 12efdc2 merging
new 79a96d0 merging
new b2a1a29 Merge pull request #34 from shintoio/jetty_http_server
new e2a3b5c Changed amaterasu path
new c049593 ast dumps tree
new 9adeda0 executing python code with no output
new 31fc491 python execution test passes
new ec7bc83 Fixed merge issue
new 7da28c6 Fixed merge issue
new f92014f python errors are now handled
new 4abc82a pyspark works!
new 330873f unicode issue :(
new 73ceca9 still not working :\
new b4af206 fixed netty issue
new fec3df3 fixed netty issue again
new c04f1f8 converted JavaSparkContext instance consumed by the script into a patched version to conform to pyspark's internals.
new bf3c129 SparkSql support added
new fd9cd1b python errors are fixed, issue with netty
new 0d5d00f Auto-Detects and reads JSON and PARQUET data sources irrespective of whether the given data source is a directory or a file. CSV support will be added in the future versions
new a4dd714 added pythonpath
new 8053fee simple python tests work
new 3d5eefc pyspark tests with ama_context work
new 7ce642d fixed tests and feedback from Eyals code review
new c1b142c Merge branch 'master' into pyspark-support
new 7bd2f6d Merge pull request #37 from shintoio/pyspark-support
new 0b221e4 Merge branch 'master' into sparksql-support
new ec9478e Merge pull request #35 from shintoio/sparksql-support
new f6d6a79 refactoring + move to gradle, all tests pass
new 169d8c4 refactored common and executor packages
new f778916 leader package refactoring
new aa21807 added shadowJar back
new dc4d8e4 buildHomeDir gradle task added
new 3eaa697 creates a valid home dir
new 6bcb82b added buildHomeDir and buildDistribution tasks
new 35e510b using the new provider factory to dynamically load runners
new 4bfce01 pyspark and py4j introduce a lot of python code that shouldn't exist permanently for the sole purpose of running tests
new 8e7da40 Merge pull request #39 from roadan/techcard/refactoring
new 3898e75 Merge pull request #40 from roadan/feature/select-runner-by-name
new c4e752e fixed path in python tests
new 55293e1 Merge pull request #41 from roadan/bugfix/tests
new 49fc9fd unit tests are passing, issue when runing on the cluster
new d59be78 spark io issue
new 7f4b68b added initial exports support
new 6d49ccf fixed spark.repl.class.uri, still doesnt work
new 0f80419 fixed sparkconf (hard-coded)
new 94877ae - aligned jre version to 1.8 in all projects - aligned scala version to 2.11.11 - got rid of SQLContext in SparkSqlRunner - endless attempts (and covfefe attempts) to reconcile ClassCastException I believe this is due to spark 2 changes in the interpreter implementations see: https://issues.apache.org/jira/browse/SPARK-20525
new 9224320 moved to spark 2.1.1
new 8a6ce1a spark 2 working!
new 159f435 spark 2 working!
new 199db93 pyspark 2 tests are now building still not running
new 85f18c8 conf values are transfered to executor and loaded by the provider
new ec0e7de Merge pull request #43 from roadan/feature/remove-test-py-folders
new ae3119e spark configuration is added to conf, still not effecting the container for the driver
new b174b43 tests are still braking
new e3237b7 all is YAML
new 485f3cb tests are passing and added suppor for executor environment variables
new 65396ba changed spark_exec to spark_exec_env
new 703e63a merged pr/42
new dad5a9f Merge branch 'master' into feature/provider-configuration
new 12cb358 Merge pull request #44 from roadan/feature/provider-configuration
new 62638ad fixed tests and resources after merge
new 7089e80 fixed tests and resources after merge
new ce41740 fixed merge bug
new 078e9b5 added get_dataframe and env to python runtime
new 1b547fc added ama_context methods and env
new b1a7fb3 Begin of Apache code donation and refactoring
new 71fb481 Refactore Amaeterasu SDK
new f442e03 Merge pull request #46 from jbonofre/APACHE_DONATION
new 33c866c pyspark ama_context working and tested
new 5766638 added get_dataframe and env to python runtime
new 15f6700 added ama_context methods and env
new f37d103 pyspark ama_context working and tested
new 5ef0358 rebased to the apache commit
new bf7b8b4 cleanup
new 3f60029 cleanup
new da0d2af cleanup
new b0397ee cleanup
new 9754e97 fixed string references to shointo
new 0b6cb06 Python 2 classes should inherit from object
new 546e36e Python 2 classes should inherit from object
new e91841d Merge pull request #47 from roadan/feature/python-runtime
new 99ee0f1 fixed path in python tests
new 08e3c48 pyspark and py4j introduce a lot of python code that shouldn't exist permanently for the sole purpose of running tests
new 26ecf9e unit tests are passing, issue when runing on the cluster
new 26a2475 spark io issue
new 0c5bf24 added initial exports support
new 527526a fixed spark.repl.class.uri, still doesnt work
new a1ae6ca fixed sparkconf (hard-coded)
new 08a7f79 - aligned jre version to 1.8 in all projects - aligned scala version to 2.11.11 - got rid of SQLContext in SparkSqlRunner - endless attempts (and covfefe attempts) to reconcile ClassCastException I believe this is due to spark 2 changes in the interpreter implementations see: https://issues.apache.org/jira/browse/SPARK-20525
new e62068e moved to spark 2.1.1
new 5ecd1da spark 2 working!
new 4ec30ca spark 2 working!
new 3eed7cd pyspark 2 tests are now building still not running
new bcac75d fixed tests and resources after merge
new 9fde5dd conf values are transfered to executor and loaded by the provider
new f2c4b65 spark configuration is added to conf, still not effecting the container for the driver
new a16327a tests are still braking
new 06a7e04 all is YAML
new e3cfc34 tests are passing and added suppor for executor environment variables
new b9b6db0 changed spark_exec to spark_exec_env
new 0ddeab1 fixed merge bug
new 3775a26 Begin of Apache code donation and refactoring
new 4e51fa5 Refactore Amaeterasu SDK
new 49e5c58 added get_dataframe and env to python runtime
new 7941e8f added ama_context methods and env
new c2b86c1 pyspark ama_context working and tested
new 611bb6f rebased to the apache commit
new ab234e7 cleanup
new 93c0735 pyspark ama_context working and tested
new 32869fd cleanup
new f03944d cleanup
new 9162a35 fixed string references to shointo
new 3a826ce Python 2 classes should inherit from object
new 7bc9a7d Python 2 classes should inherit from object
new f3f00f8 moving over changes form the corrupt branch manually
new b96a120 moving over changes form the corrupt branch manually
new 4f72ce6 moving over changes form the corrupt branch manually
new cb716c4 moving over changes form the corrupt branch manually
new 3edd9f8 copy python resources
new 16637f7 copy python resources
new 472a3ac copy python resources
new a048a8d copy python resources
new e663c07 changed dependencies format
new 6149b71 logging
new 667c442 logging
new 283939c logging
new fb84521 logging
new c16729f logging
new 7875a71 logging
new 581cc9e logging
new 80b8f73 logging
new 90097e2 logging
new 6b603c7 logging
new 0b83738 logging
new 13daaa4 logging
new dc15e35 spark 2.0 uses a different executable
new 1bcc071 corrected spark_intp and other good things
new 4c37c3a corrected spark_intp and other good things
new 3f682b9 tried to open a file, it is wrong, should pass the source as it is.
new 2ca80e3 Merge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi/amaterasu into feature/conda-dependencies-fixed
new cf40811 seems wrong?
new 7bf1701 trying to figure out why scala runner is throwing null
new aae0f69 trying to figure out why scala runner is throwing null
new 0cc1045 removed notifier calls
new e25aa7f non-conda code fails
new ba29e48 documentation
new 37e9713 started fitting tests and trying to solve an issue with Pyspark not recognized
new 11edcde missed passing pythonpath to actual executed script
new d6e1f6a cleaned up
new 03dca7e trying to make this work by configuration, so tests can actually work
new 2b63d25 Merge branch 'feature/conda-dependencies-fixed' of https://github.com/nadav-har-tzvi/amaterasu into feature/conda-dependencies-fixed
new ead626d pyspark tests almost work, exports fail
new ad1c2c0 duh
new e17ad3d hardcode it for now, unless someone wants to control this
new 74c6432 still couldn't catch this, no idea why
new 0ee62cc wrong one...
new 1049d02 test tried writing into the fs root instead of the CWD
new a531903 write test exports to /tmp to get rid of them
new 6484e53 forgot to add pyspark.zip
new 9c19296 trying to inject factory
new 9a835c0 fixed double instanciation of tests
new cc7960e cleanup and tests passing
new bda9b31 Merge branch 'nadav-har-tzvi-feature/conda-dependencies-fixed'
new 4751974 Merge branch 'ama-master' into feature/conda-dependencies-fixed
new d22637f Merge branch 'ama-master' into feature/conda-dependencies-fixed
new d4a05e5 Merge pull request #48 from nadav-har-tzvi/feature/conda-dependencies-fixed
The 288 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
--
To stop receiving notification emails like this one, please contact
['"commits@amaterasu.apache.org" <co...@amaterasu.apache.org>'].