You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@wayang.apache.org by be...@apache.org on 2021/03/05 16:30:13 UTC
[incubator-wayang] branch python-api updated (ff0cb9f -> 828c9c2)
This is an automated email from the ASF dual-hosted git repository.
bertty pushed a change to branch python-api
in repository https://gitbox.apache.org/repos/asf/incubator-wayang.git.
from ff0cb9f - Changed the Jenkinsfile to deploy for "main" not "develop"
new 98883bf [WAYANG-8][API-PYTHON] Creation of functions to be consumed by MapPartitionsDescriptor
new 545247e [WAYANG-8][API-PYTHON] Included PythonProcessCaller that manages the python process execution and Java - Python connection
new a47a9ee Merge with origin/main
new 64b405e [WAYANG-8][API-PYTHON] POM fixes plus minor test
new b717e63 [WAYANG-8][API-PYTHON] Python connection through TCP socket enabled
new 19616e6 [WAYANG-8][API-PYTHON] Writing from Java to Python. Not taking into care about Iterator Datatypes.
new e99a02d [WAYANG-8][API-PYTHON] Java Socket Writter improvements
new 69c3c59 [WAYANG-8][API-PYTHON] Python UTF8 Deserializer included
new 424cc22 [WAYANG-8][API-PYTHON] Python UTF8 Reading Stream
new 105d8de [WAYANG-8][API-PYTHON] Getting results from Python and continue processing
new b6b7140 [WAYANG-8][API-PYTHON] Config files for pywayang
new f3b8e03 [WAYANG-8][API-PYTHON] Structures to save the plan with functional fashion plus most basic operators
new 3a5dea6 [WAYANG-8][API-PYTHON] Main program to test plan executions locally
new be1e033 [WAYANG-8][API-PYTHON] Minor comments and TODOs
new 2a3880c [WAYANG-8][API-PYTHON] Most basic test for protobuff communication with java
new 4203750 [WAYANG-8][API-PYTHON] Addjacency list from PyWayang Plan
new 7e2c608 [WAYANG-8][API-PYTHON] Graph traversal implementation with visitor pattern
new 4c9b544 [WAYANG-8][API-PYTHON] Protobuf python message generator
new caca926 [WAYANG-8][API-PYTHON] Wayang Web Service project structure
new 753ed3d [WAYANG-8][API-PYTHON] Protobuf message generation fixes
new 49d8346 [WAYANG-8][API-PYTHON] Wayang Web Service executes most basic plans directly
new 315acad [WAYANG-8][API-PYTHON] Receiving Base64 passing to byte array and unpickling
new 43e3bea [WAYANG-8][API-PYTHON] Updated classes to process a single Serialized UDF
new ea4fcd1 [WAYANG-8][API-PYTHON] New test with single UDF
new bd2cea2 [WAYANG-8][API-PYTHON] Protobuf command
new 33fda4b [WAYANG-8][API-PYTHON] Protobuf message template updated
new ec97fa7 [WAYANG-8][API-PYTHON] POM fixes
new f5d957a [WAYANG-8][API-PYTHON] License comments added
new fc22a2d Merge remote-tracking branch 'upstream/main' into main
new 61aa6ca Merge remote-tracking branch 'upstream/main' into main
new 978d4fd [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the core
new 0734b25 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-basic
new fc5eaa0 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-flink
new 4337094 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-giraph
new 281edc1 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-graphchi
new 7c88a68 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-java
new 13b198c [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-jdbc-template
new 5b231c1 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-spark
new 9549fc6 [WAYANG-3][NOT-COMPILE] change the logs library to Log4J2 in the wayang-profiler
new 46117ca [WAYANG-3] change the logs library to Log4J2 minnor changes
new 449e868 [WAYANG-12] creation of the wayang-resource and modification of the parent pom
new 4c93999 [WAYANG-12] add profile build-doc and modificatin in distro distro
new a57eb67 [WAYANG-12] travis correction of jdk
new d04bd90 [WAYANG-12] showing error rats
new 4c4884c [WAYANG-12] add license to travis/wayang.properties is an temporal file
new bdf2305 [WAYANG-12] add cache maven dependecies in travis
new e305b36 [WAYANG-12] not found class error
new 6cfa254 [WAYANG-12] removing messages that are not need it
new f401cad [WAYANG-12] hidding output to file and showing the last 2000 lines
new 276b5b9 [WAYANG-12] deletiong of the rat file
new 9fbc16b [WAYANG-12] remove cache to test
new 4d02d86 [WAYANG-12] remove cache to test
new 75bb00d [WAYANG-12] remove cache to test
new 7535cc2 [WAYANG-12] remove cache to test
new 689fbd1 Merge branch 'main' into issues
new 0ae020c [WAYANG-12] remove test with error to do some test
new 9c78176 [WAYANG-12] comment command that does not exist
new af0aaca [WAYANG-12] remove test to validate if it run
new 969171e Merge pull request #7 from berttty/issues
new b1f1463 fix broken link / updated Apache License claim
new 96cf918 Merge pull request #8 from 2pk03/main
new 05371fb Merge remote-tracking branch 'upstream/main' into python-api
new 6fe2019 [WAYANG-8][API-PYTHON] Correction on missing licenses
new 94bf742 [WAYANG-8][API-PYTHON] Serializable module creation
new 0f0c226 [WAYANG-8][API-PYTHON] adding protoc to travis
new 47cfd7b [WAYANG-8][API-PYTHON] protoc executable path correction
new 24a55f8 [WAYANG-8][API-PYTHON] Commenting objc_class_prefix
new 828c9c2 Merge pull request #9 from ro-pardo/python-api
The 1195 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.travis.yml | 35 +++-
README.md | 6 +-
pom.xml | 47 ++----
.../pyplangenerator.sh | 4 +-
.../LruCache.java => protobuf/pywayangplan.proto | 62 ++++---
.../config/__init__.py | 4 +-
pywayang/config/config_reader.py | 51 ++++++
.../config/pywayang_config.ini | 25 ++-
.../graph/__init__.py | 4 +-
pywayang/graph/graph.py | 68 ++++++++
pywayang/graph/node.py | 44 +++++
pywayang/graph/traversal.py | 49 ++++++
pywayang/graph/visitant.py | 51 ++++++
.../orchestrator/__init__.py | 5 +-
pywayang/orchestrator/dataquanta.py | 136 +++++++++++++++
pywayang/orchestrator/main.py | 76 +++++++++
pywayang/orchestrator/operator.py | 110 ++++++++++++
.../orchestrator/plan.py | 25 ++-
.../protobuf/__init__.py | 2 +-
pywayang/protobuf/planwriter.py | 152 +++++++++++++++++
wayang-api/pom.xml | 1 +
wayang-api/wayang-api-python/pom.xml | 46 ++++-
.../wayang/api/python/executor/ProcessFeeder.java | 177 ++++++++++++++++++++
.../api/python/executor/ProcessReceiver.java | 44 +++--
.../api/python/executor/PythonProcessCaller.java | 118 +++++++++++++
.../api/python/executor/PythonWorkerManager.java | 64 +++++++
.../wayang/api/python/executor/ReaderIterator.java | 86 ++++++++++
.../{PythonAPI.java => function/PythonUdf.java} | 7 +-
.../api/python/function/WrappedPythonFunction.java | 36 ++--
.../wayang-api-python/src/main/python/worker.py | 186 +++++++++++++++++++++
.../wayang-api-python-defaults.properties | 3 +-
.../org/apache/wayang/api/python/BasicPython.java | 67 ++++++++
.../org/apache/wayang/api/python/SortPython.java | 102 +++++++++++
.../wayang-api-rest}/pom.xml | 71 ++++----
.../org/apache/wayang/api/rest/WayangRESTAPI.java} | 8 +-
.../api/rest/server/spring/WayangApplication.java | 21 +--
.../rest/server/spring/decoder/pyplanreader.sh} | 4 +-
.../server/spring/general/WayangController.java | 159 ++++++++++++++++++
.../src/main/resources/application.properties} | 2 +-
wayang-commons/pom.xml | 13 +-
.../wayang/basic/operators/SampleOperator.java | 6 +-
.../wayang/basic/operators/TextFileSource.java | 6 +-
.../wayang/basic/operators/TextFileSourceTest.java | 7 +-
wayang-commons/wayang-core/pom.xml | 8 +
.../org/apache/wayang/core/api/Configuration.java | 6 +-
.../main/java/org/apache/wayang/core/api/Job.java | 6 +-
.../org/apache/wayang/core/api/WayangContext.java | 6 +-
.../core/api/configuration/CollectionProvider.java | 6 +-
.../configuration/ExplicitCollectionProvider.java | 6 +-
.../core/api/configuration/KeyValueProvider.java | 6 +-
.../core/api/configuration/ValueProvider.java | 6 +-
.../wayang/core/mapping/PlanTransformation.java | 6 +-
.../wayang/core/optimizer/OptimizationContext.java | 6 +-
.../wayang/core/optimizer/SanityChecker.java | 6 +-
.../cardinality/CardinalityEstimatorManager.java | 6 +-
.../optimizer/cardinality/CardinalityPusher.java | 6 +-
.../SwitchForwardCardinalityEstimator.java | 4 +-
.../optimizer/channels/ChannelConversionGraph.java | 6 +-
.../optimizer/costs/LoadProfileEstimators.java | 6 +-
.../optimizer/enumeration/ExecutionTaskFlow.java | 6 +-
.../enumeration/ExecutionTaskFlowCompiler.java | 4 +-
.../enumeration/LatentOperatorPruningStrategy.java | 10 +-
.../core/optimizer/enumeration/LoopEnumerator.java | 4 +-
.../optimizer/enumeration/PlanEnumeration.java | 6 +-
.../core/optimizer/enumeration/PlanEnumerator.java | 6 +-
.../optimizer/enumeration/PlanImplementation.java | 6 +-
.../enumeration/StageAssignmentTraversal.java | 6 +-
.../wayang/core/plan/executionplan/Channel.java | 8 +-
.../core/plan/executionplan/ExecutionPlan.java | 6 +-
.../core/plan/wayangplan/ExecutionOperator.java | 4 +-
.../wayang/core/plan/wayangplan/LoopIsolator.java | 4 +-
.../core/plan/wayangplan/OperatorContainer.java | 4 +-
.../wayang/core/plan/wayangplan/PlanTraversal.java | 6 +-
.../wayang/core/plan/wayangplan/SlotMapping.java | 6 +-
.../wayang/core/plan/wayangplan/WayangPlan.java | 6 +-
.../traversal/AbstractTopologicalTraversal.java | 6 +-
.../core/platform/AbstractChannelInstance.java | 4 +-
.../core/platform/CardinalityBreakpoint.java | 6 +-
.../core/platform/CrossPlatformExecutor.java | 6 +-
.../wayang/core/platform/ExecutorTemplate.java | 6 +-
.../org/apache/wayang/core/platform/Junction.java | 6 +-
.../platform/lineage/LazyExecutionLineageNode.java | 4 +-
.../core/profiling/CardinalityRepository.java | 6 +-
.../apache/wayang/core/profiling/ExecutionLog.java | 6 +-
.../core/util/AbstractReferenceCountable.java | 7 +-
.../java/org/apache/wayang/core/util/Action.java | 4 +-
.../org/apache/wayang/core/util/JuelUtils.java | 6 +-
.../java/org/apache/wayang/core/util/Logging.java | 6 +-
.../apache/wayang/core/util/ReflectionUtils.java | 6 +-
.../apache/wayang/core/util/fs/FileSystems.java | 8 +-
.../wayang/core/util/fs/HadoopFileSystem.java | 10 +-
.../wayang/core/util/fs/LocalFileSystem.java | 6 +-
.../core/util/ConsumerIteratorAdapterTest.java | 6 +-
wayang-commons/wayang-serializable/pom.xml | 80 +++++++++
.../src/main/proto/pywayangplan.proto | 73 ++++++++
.../flink/compiler/WayangFileOutputFormat.java | 6 +-
.../wayang/flink/platform/FlinkPlatform.java | 6 +-
.../wayang/giraph/Algorithm/PageRankAlgorithm.java | 7 +-
.../giraph/operators/GiraphPageRankOperator.java | 6 +-
.../operators/GraphChiPageRankOperator.java | 6 +-
.../wayang/java/operators/JavaObjectFileSink.java | 4 +-
.../java/operators/JavaObjectFileSource.java | 4 +-
wayang-platforms/wayang-jdbc-template/pom.xml | 17 ++
.../apache/wayang/jdbc/execution/JdbcExecutor.java | 6 +-
.../wayang/jdbc/operators/JdbcTableSource.java | 4 +-
.../wayang/jdbc/operators/SqlToStreamOperator.java | 4 +-
.../spark/operators/SparkObjectFileSink.java | 4 +-
.../spark/operators/SparkObjectFileSource.java | 6 +-
.../wayang/spark/platform/SparkPlatform.java | 6 +-
.../wayang/profiler/java/OperatorProfiler.java | 6 +-
.../wayang/profiler/java/SourceProfiler.java | 5 +-
.../profiler/log/DynamicLoadProfileEstimators.java | 6 +-
.../wayang/profiler/log/GeneticOptimizerApp.java | 6 +-
.../apache/wayang/profiler/log/LogEvaluator.java | 6 +-
.../profiler/spark/SparkOperatorProfiler.java | 6 +-
.../wayang/profiler/util/ProfilingUtils.java | 4 +-
.../apache/wayang/profiler/util/RrdAccessor.java | 6 +-
{wayang-api => wayang-resources}/pom.xml | 39 +++--
.../src/main/assembly/resources.xml | 39 ++---
.../src/main/resources/assemblies/resources.xml | 43 ++---
.../apache/wayang/tests/GraphChiIntegrationIT.java | 12 +-
121 files changed, 2447 insertions(+), 457 deletions(-)
copy wayang-api/wayang-api-python/src/main/resources/wayang-api-python-defaults.properties => protobuf/pyplangenerator.sh (77%)
copy wayang-commons/wayang-core/src/main/java/org/apache/wayang/core/util/LruCache.java => protobuf/pywayangplan.proto (53%)
copy wayang-api/wayang-api-python/src/main/resources/wayang-api-python-defaults.properties => pywayang/config/__init__.py (83%)
create mode 100644 pywayang/config/config_reader.py
copy wayang-platforms/wayang-jdbc-template/src/test/resources/wayang-hsqldb-defaults.properties => pywayang/config/pywayang_config.ini (72%)
copy wayang-api/wayang-api-python/src/main/resources/wayang-api-python-defaults.properties => pywayang/graph/__init__.py (90%)
create mode 100644 pywayang/graph/graph.py
create mode 100644 pywayang/graph/node.py
create mode 100644 pywayang/graph/traversal.py
create mode 100644 pywayang/graph/visitant.py
copy wayang-extensions/wayang-iejoin/src/test/resources/wayang.properties => pywayang/orchestrator/__init__.py (87%)
create mode 100644 pywayang/orchestrator/dataquanta.py
create mode 100644 pywayang/orchestrator/main.py
create mode 100644 pywayang/orchestrator/operator.py
copy wayang-platforms/wayang-jdbc-template/src/test/resources/wayang-hsqldb-defaults.properties => pywayang/orchestrator/plan.py (62%)
copy wayang-extensions/wayang-iejoin/src/test/resources/wayang.properties => pywayang/protobuf/__init__.py (95%)
create mode 100644 pywayang/protobuf/planwriter.py
create mode 100644 wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/executor/ProcessFeeder.java
copy wayang-commons/wayang-core/src/main/java/org/apache/wayang/core/platform/NoIterationBreakpoint.java => wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/executor/ProcessReceiver.java (51%)
create mode 100644 wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/executor/PythonProcessCaller.java
create mode 100644 wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/executor/PythonWorkerManager.java
create mode 100644 wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/executor/ReaderIterator.java
copy wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/{PythonAPI.java => function/PythonUdf.java} (77%)
copy wayang-platforms/wayang-jdbc-template/src/main/java/org/apache/wayang/jdbc/compiler/FunctionCompiler.java => wayang-api/wayang-api-python/src/main/java/org/apache/wayang/api/python/function/WrappedPythonFunction.java (52%)
create mode 100644 wayang-api/wayang-api-python/src/main/python/worker.py
create mode 100644 wayang-api/wayang-api-python/src/test/java/org/apache/wayang/api/python/BasicPython.java
create mode 100644 wayang-api/wayang-api-python/src/test/java/org/apache/wayang/api/python/SortPython.java
copy {wayang-profiler => wayang-api/wayang-api-rest}/pom.xml (59%)
copy wayang-api/{wayang-api-python/src/main/java/org/apache/wayang/api/python/PythonAPI.java => wayang-api-rest/src/main/java/org/apache/wayang/api/rest/WayangRESTAPI.java} (89%)
copy wayang-commons/wayang-core/src/main/java/org/apache/wayang/core/util/Actions.java => wayang-api/wayang-api-rest/src/main/java/org/apache/wayang/api/rest/server/spring/WayangApplication.java (71%)
copy wayang-api/{wayang-api-python/src/main/resources/wayang-api-python-defaults.properties => wayang-api-rest/src/main/java/org/apache/wayang/api/rest/server/spring/decoder/pyplanreader.sh} (72%)
create mode 100644 wayang-api/wayang-api-rest/src/main/java/org/apache/wayang/api/rest/server/spring/general/WayangController.java
copy wayang-api/{wayang-api-python/src/main/resources/wayang-api-python-defaults.properties => wayang-api-rest/src/main/resources/application.properties} (96%)
create mode 100644 wayang-commons/wayang-serializable/pom.xml
create mode 100644 wayang-commons/wayang-serializable/src/main/proto/pywayangplan.proto
copy {wayang-api => wayang-resources}/pom.xml (54%)
copy wayang-api/pom.xml => wayang-resources/src/main/assembly/resources.xml (52%)
copy wayang-api/pom.xml => wayang-resources/src/main/resources/assemblies/resources.xml (50%)