You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by mm...@apache.org on 2018/03/04 03:59:53 UTC
[incubator-pulsar] branch master updated (44e0663 -> 12dac7e)
This is an automated email from the ASF dual-hosted git repository.
mmerli pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git.
from 44e0663 Utilities for arquillian integration tests (#1310)
add 72092c4 Create pulsar-functions module (#1)
add 9a56774 Added the first cut of the Java interface for Pulsar functions (#2)
add a13d5ce Instance (#3)
add 468c6e1 Only test pulsar-functions (#5)
add 9e7d30c Use TypeResolver to resolve generic types (#7)
add 7c871b0 Make RequestHandler a functional interface (#8)
add aca6ac5 Introduce Function Config and Functions Cmd (#4)
add c1b68ed Run travis ci on linux and osx (#10)
add 920277b Introduce `functioncache` and `functioncontainer` (#9)
add e30808c Spawner (#11)
add 5e76557 Bugcix (#12)
add d2b4483 Split functions module into api/runtime/api-examples (#13)
add bb6409f Add examples (#14)
add da4957b Fix bugs (#15)
add e6978ba Integrate CLI with spawner to run functions locally (#16)
add 98d208d Add the missing `pulsar-function` script
add 5b0ed1b Minor stuff (#17)
add 52c9c24 Refactor (#18)
add 85e1882 Simplestringserde (#19)
add 228e01b Fix broken license headers at pulsar-functions (#20)
add 9766676 Opt (#22)
add fc1f71c Cleanup (#23)
add 5f3b93e Fix tests (#24)
add f883f16 Buffer limit (#25)
add 5adc066 Relocate packages (#21)
add 117f816 System exception (#26)
add f026ef8 Refactor cli (#27)
add 13bbbf7 Use the thread context classloader for loading functions (#28)
add 316f834 relocate SerDe classes to `serde` package and add tests (#29)
add 275daf0 Make SerDe interface a generic interface and load the serde class in JavaInstance (#30)
add a548bc2 Input and Output Serdes (#31)
add 4dd4089 Nomenclature (#32)
add 48d4aa4 Added Submit Function capability to the pulsar-functions cli (#33)
add 3227065 Cleanup (#34)
add da66b2d Fix compilation (#35)
add 3f7d304 Make producer and consumer async
add 7fbe342 Asyncack (#37)
add e90b308 Nonthreadpool (#38)
add 5c1e369 Provide a jmh benchmark to benchmark thread container (#39)
add 4b4bea7 Reconcile (#40)
add a1b1ea5 Made Update a put request (#41)
add 27f4c0b Enable batching on producer (#42)
add 5077d71 Completecli (#43)
add 0444ac4 Introduce FunctionStats to collection function related stats (#44)
add 5ea8065 Fileinterface (#45)
add 5c4b286 Print (#46)
add 570d59c Initial implementation of Pulsar Function Worker (#47)
add 103d20d Fix license header (#48)
add 8824efb Compose `FunctionState` with FunctionConfig and LimitsConfig (#49)
add 84a8040 Load function worker configuration from yaml file (#50)
add 0c113d7 Rewrite FunctionStateListener to use receiveAsyc (#51)
add 59ec630 Rename functionstate (#52)
add 793b1cf Renamed FunctionState stuff to something that reflects their functionality (#53)
add d0c231e Refactor to support at-least-once processing (#54)
add 7c10a31 Cleanup Cli commands (#55)
add eefe67e Invoke spawner upon start/stop of worker (#56)
add 58a5392 Removed ExecutionResult since its not longer needed (#57)
add 507b938 fixing log functions (#58)
add 4f107d3 Use LinkedBlockingQueue#put and ack before processing for at-most-once processing (#59)
add 4c973c8 Added atmost once and atleast once semantics (#60)
add 38a5987 Make sure it works if the function does not have a sink topic (#61)
add 43a09e6 add/fixes the following: (#62)
add 9902a32 Moved FunctionStats to fs since it does not belong to api (#64)
add ef7657c Organize function packages in hierarchy (#65)
add c74e545 Removing ServiceRequest from requestResult (#66)
add d14c23a Added FunctionActioner to start/stop functions in seperate thread (#68)
add 61650f3 Add unit tests for dlog classes and rest api (#67)
add 5fe559d Add the missing script for running pulsar worker (#69)
add 8a0cd54 Added more logging. Also changed getFunctions to return 404 when not found (#71)
add 4e6adca Add `@RequiredField` annotation to enforce verification on whether a field is set or not. (#73)
add 7e14c8f More graceful warning when args are missing (#72)
add f1081f8 add client side class validation (#70)
add 2dc4a5b Bug fix and add log messages (#75)
add b662fe6 Function Logging (#74)
add 992e947 Name our threads (#76)
add 46d8874 Revert dlog version to stable 0.5.0 (#77)
add 638e3c7 Fix spelling (#78)
add 6a4592a fixing bugs and adding assignments datastructure (#79)
add 8002e3b Use Reader API instead of Consumer API (#81)
add 2fc20b5 Upgrade testng and downgrade log4j2 and remove junit (#80)
add ee0ba86 Fix upload path issue (#83)
add ab3007f Removed FunctionStats interface and added FunctionStatus class (#82)
add 5dcf03a If the function is not found, send 404 (#84)
add aa19226 Pulsar worker uses routing appender (#85)
add d5955d8 adding initialize routine to launch existing assignments (#86)
add f4130f8 Added getstatus function (#87)
add 5650574 Refactor worker to make sure that FunctionActioner is just an actioner. Also consolidate all info in worker in FunctionRuntimeInfo (#88)
add 1b0607a Added ability to propagate user defined configs (#89)
add d82a04f Move worker module out of runtime module (#90)
add 71997bb Make ThreadFactory as thin as possible (#91)
add 493e54a Made use of LimitsConfig in JavaInstanceConfig (#92)
add f65c4a4 Move most of the instance logic to JavaInstanceRunnable (#93)
add 72a9804 Moved FunctionId/InstanceId to fs package (#94)
add 9abeef2 Simplify Container interface (#95)
add fe80a6c adding generated code module (#96)
add d07897a updating generated code dir (#97)
add ce81cb7 fixing generated dir (#98)
add 426ab54 Migrating ServiceRequest to proto (#99)
add 7b24f68 Removed FunctionId and InstanceId (#100)
add 8316228 Refactor functions module structure (#101)
add a0e4204 Rename generated module to proto module (#102)
add ec46e1a LimitsConfig is always part of WorkerConfig (#103)
add d02ae48 Migrating FunctionConfig, FunctionMetaData, and ServiceRequest to proto (#104)
add cb74f21 Added ProcessContainerFactory (#105)
add 7ea376a Removed FunctionStatus and use protobuf as the central datastructure (#106)
add d400a03 Added Metric Interface (#107)
add 1c4bbaf Added metrics sinks interface and added file and prometheus interface (#108)
add 5bf96a7 Added ability to specify metrics sinks (#109)
add 674b2a2 Misc bug fixes to get metrics to work (#110)
add 45c807e Added system metrics to the list of metrics collected (#111)
add 8d4ff95 Shade `pulsar-client-admin` (#112)
add 4cbec31 adding log4j confs for unit tests (#114)
add 882d56f Adding snapshot feature to Pulsar Functions worker (#113)
add 8fcdd09 Refactor serde classes into JavaInstanceRunnable (#115)
add 693ae3b Use `pulsar-client-tools-shaded` jar to avoid inconsistency between shaded jars and avoid shading jackson (#116)
add 2cb75ba Fixing response and cli (#117)
add ba06963 Added ability for a function to consume multiple input topics (#118)
add 2e438b5 Add `join` interface to container to wait for a container (#121)
add 28ff628 Use a different location for caching the classpath (#122)
add 5dfbb27 Move SerDe to api (#120)
add b3a8b21 Added publish interface to the api (#123)
add 65f02b3 Moved JavaInstanceConfig to InstanceConfig (#125)
add f2d511e Added ability to specify subscription type (#124)
add 6c380c4 Added runtime descriptor in functionConfig (#126)
add f50a68e adding additional worker tests (#127)
add 850c1bd Moved api/examples to api-java and java-examples (#128)
add 0e84121 Removed RawRequest interface (#130)
add 624afe7 Renamed RequestHandler to PulsarFunction (#131)
add 8f571de Added python api (#132)
add b590e8b Add plumbing to accept python inside worker/container (#133)
add 0863c22 adding user acking and fixing bugs (#134)
add fea9805 First cut of Python Instance that just parses the arguments and start… (#135)
add 6f853cd First of cut of windowing for Pulsar Functions (#136)
add 0ee69b9 renaming to streamlet (#138)
add 0667a47 Provide a copy task in runtime package phase to copy all python-instance required files into `target/python-instance` (#137)
add d2f32e8 Feature Complete Python Instance. (#139)
add a2ed4c2 Kill process upon shutdown (#140)
add f1c403c Quit cleanly upon ctrl-c (#141)
add bc074c8 Propagate Auto Acking information to the processes (#142)
add 937b57d Support explicit acking in the context just like Java interface (#143)
add 7af9d02 Moved Stats to java instance since it now only has visibility in java world only (#144)
add 331e658 Event time windowing for Pulsar Functions (#145)
add 5241991 Remove calculating of quantiles and keep stats simple (#146)
add f77cd00 moving windowing code (#147)
add 1ebc04d Overhaul stats in both Java and Python instances (#148)
add caddfd7 Periodically check the liveness of the function process (#149)
add 3682ed0 adding unittest for windowing (#150)
add 4b24eb6 Correct spacing (#151)
add 907b6e4 changing publish function to accept string for classname (#152)
add 27cb857 Added scripts to generate python packages (#153)
add 86c8897 correcting header files (#154)
add a94e2d8 remove possible race condition (#156)
add f0d9e1d Convert user config to json in process builder (#157)
add 5f55daa Remove type checking since Serde is now exposed (#159)
add 6e8d14e localrun uses broker admin url and remove empty module (#160)
add 4a82004 Removed the usage of timer since it doesnt seem to be used (#161)
add 21aec14 State - Counter support (#155)
add 46fe5d2 Interrupt instance thread when stopping function container (#162)
add 3f963d8 Added SimpleSerDe to ser/deser basic types (#163)
add 4396db6 Make state storage service url (#164)
add 989f59d Improve pulsar functions logging (#165)
add 2f6167e Improve the logic for finding java instance location (#166)
add efce272 Use Spawner inside Java Instance rather than using JavaInstance directly (#169)
add edbae3c Allow two types of input. (#170)
add 5f84eb1 Make Output Serde Optional (#171)
add 3424041 Infer Missing Arguments (#172)
add 0e4a4da Add more logs while starting thread container (#173)
add bcecfcc More checks for checking the presence of certain fields (#174)
add 06e7d74 Change the log file name to the name of the function. This greatly as… (#175)
add 3b7d1f4 Return a valid response upon not being able to get status from spawner (#176)
add d498fa9 Move SimpleSerDe to api (#177)
add 138da83 Simplify DefaultSerde and make users be able to explicitly set it (#178)
add 14b0fe4 Removed Utf8SerDe now that we have its more generic alternative DefaultSerDe (#179)
add 75ce665 Python Instance should resort to default serializer if nothing is specified (#180)
add 467f6e4 Use the right regexp for extracting class name (#181)
add ecf223c Assume a default output topic if none is specified and create one (#182)
add aa36dcb Fix python location (#168)
add 10e45d6 Added more python examples and fix existing (#183)
add a954d96 Since server is already module name, rename the variable (#185)
add 3448c10 during startup self.producer will always be null (#186)
add 180f01f Implementing distributed worker mode (#184)
add dfb333f Added a mechanism to configure user config via cli as well (#187)
add 79a3b78 Fixing Function get status (#188)
add 555a08b Add more type checks for Java submissions (#189)
add 0c7e7b5 Check for void type for output serde (#190)
add ef5f986 DefaultSerDe needs to be init specially for user publish (#191)
add 37a0b63 Added Java Native Functions capability (#192)
add a025bfe Added support for Pure Python Functions (#193)
add dd8b4a8 Raise an exception if the module is not found (#194)
add dfcfe5f Avoid race condition by making methods synchronized (#195)
add ac5ca04 adding check for failures subroutine and fixing worker delete bug (#196)
add 922494e Correct the order of checking (#197)
add 868d21d Reload class for the function/serde type checking on submitting functions (#199)
add 6779f3a synchronizing some methods and add comments (#201)
add 8df9d54 checking synchronization and adding comments (#202)
add 631f22f Consolidate error handling to Spawner (#203)
add a9c2437 Add more information about function in getstatus (#205)
add 3af1d48 fixing bug in snapshot manager (#204)
add 1832b52 fixing worker namespace initialize race condition (#206)
add 4b730b1 GetStatus should return a set of recent exceptions (#207)
add f11f192 Since we assuming that user key/value are all strings, explicitly transform result to str (#208)
add 911d289 Fix variable name (#210)
add c08b392 Always download because the code might have changed (#209)
add 0d44440 Fix the way to import pulsarfunction root specific modules (#211)
add 6fc757e Add a simpler way to publish (#212)
add b64644c Accomodate native functions (#213)
add 4b62d14 Don't count first start as a restart (#214)
add 2dccc85 fixing bug in scheduler (#215)
add 7fd924f Rename FunctionConfig names to make them more consistent (#216)
add 43ec19b Added last invocation time to the function status (#217)
add 21212a1 Added subscription type to function command (#218)
add d21341c Add a cli command to query function state (#200)
add 1bb2ccf Avoid creating pulsar admin client for local run (#219)
add 0c94f29 Removed LimitsConfig (#222)
add d1699b2 Removed timeout since it is no longer tracked (#223)
add 4b89453 Allow workers to deploy multiple function instances (#221)
add e724141 For FunctionsCommand make use of required fields (#224)
add adcdf68 adding checks for parallelism (#225)
add 0393713 adding instance id to get status (#226)
add fe1bb68 Bump functions to 2.0.0-incubating-SNAPSHOT after rebase
add e47af0c Remove zookeeper from dependency management to avoid pulling in a wrong version of zookeeper (#227)
add 8f1f022 Log before quitting (#228)
add f6b914d Address netty version conflicts causing getFunctionStatus to fail (#229)
add f8152c8 Remove zkServers from worker config (#232)
add fe6747b Get rid of the leader election trick and use ConsumerEventListener for electing a leader (#231)
add 64d2b18 Fix NPE on serializing MembershipManager (#235)
add e3403c5 adding worker retry if cannot connect to broker (#237)
add 27ad9c7 Use metadata field for worker info (#238)
add 8a99e50 Effectively once support (#230)
add cb48b09 Move all the components required for running worker (except rest endpoint) to WorkerService (#239)
add 587da92 Revert "Move all the components required for running worker (except rest endpoint) to WorkerService (#239)" (#241)
add 6b0e087 fixing getting assignments from assignment topic (#240)
add 914179c Move all the components required for running worker (except rest endpoint) to WorkerService (#242)
add 7da9755 Ignore "javax.ws.*" from PowerMockito (address rebase issue)
add f90c4d3 Upgrade state storage client to latest version (#245)
add b1306d6 Cleanup WorkerService (#243)
add fb5e34b Move function metadata setup logic out of WorkerService (#244)
add 753155d Remove snapshotting (#246)
add 2e296ee Bump log4j2 to 2.10.0
add d5285e2 Remove hardcoded jackson version (#247)
add 86f6cdc Refactored instance related stuff into instance as well as rejiggered utils into its own module (#248)
add 2d7a7f8 Refactored all metrics stuff into its own module (#249)
add 0497368 Renamed all container stuff to runtime (#250)
add ee77f77 Address merge conflicts
add 599077a Corrected the name of the java instance main (#252)
add 41c9713 Address conflicts after merged
add 2412f80 Added more comments to the JavaDoc (#253)
add bbc1242 revert zookeeper dependency
add b7d3083 Upgrade dlog deps (#254)
add 9bc41ab Increase bookkeeper disk usage threshold for pulsar standalone (#255)
add ab93a18 Update threshold (#256)
add 2fe4a38 Fix NPE for void functions (#258)
add 7370fbc Added abiltiy to specify broker on cmdline for localrun (#259)
add dc8f39c fix bug in scheduler (#260)
add 9114ddc Moved localrun to process based runtime (#261)
add e5ed2ed Provide a distribution package for pulsar functions (#263)
add e417c57 Support parallelism parameter while running functions locally (#262)
add 68e1f3c Fix pulsar-functions on locating java & python instance files (#264)
add 981e472 Use `distributedlog-core-shaded` in pulsar worker (#257)
add 6652c5a Move the thread construction to start to avoid illegalstate (#266)
add 322f3af Fix serviceUrl for functions cli (#267)
add c350e1d Improve function package download process (#268)
add a68fb0d Organize the function packages in tenant/namespace/function hierarchy (#269)
add 82bccd1 Resolve rebase conflicts
add ae1ac31 Build classpath using `compile` scope (#272)
add 2e49646 Create ProcessBuilder at during start() so that start() can be called… (#273)
add 6ec05f9 If no brokerUrl is specified use localhost (#274)
add 46bc558 fix scaling up and down as well as updating a function in general (#275)
add 03df440 adding SchedulerManager tests (#276)
add 433eb8b Better variable safety (#277)
add 5245529 Effectively once not yet supported in Python (#278)
add 12dac7e - Revert .travis.yml - Remove function-composition
No new revisions were added by this update.
Summary of changes:
pom.xml | 2 +
pulsar-functions/api-java/pom.xml | 49 +
.../org/apache/pulsar/functions/api/Context.java | 153 +++
.../pulsar/functions/api/PulsarFunction.java | 35 +
.../org/apache/pulsar/functions/api/SerDe.java | 27 +
.../apache/pulsar/functions/api/package-info.java | 23 +
.../pulsar/functions/api/utils/DefaultSerDe.java | 95 ++
.../pulsar/functions/api/utils/JavaSerDe.java | 88 ++
.../functions/api/utils/DefaultSerDeTest.java | 101 ++
.../pulsar/functions/api/utils/JavaSerDeTest.java | 57 ++
pulsar-functions/api-python/__init__.py | 21 +
pulsar-functions/api-python/context.py | 115 +++
pulsar-functions/api-python/pulsar_function.py | 52 +
pulsar-functions/api-python/serde.py | 76 ++
pulsar-functions/bin/pulsar | 192 ++++
pulsar-functions/bin/pulsar-functions | 148 +++
pulsar-functions/cli/pom.xml | 79 ++
.../org/apache/pulsar/admin/cli/CmdFunctions.java | 675 +++++++++++++
.../org/apache/pulsar/admin/cli/FunctionsTool.java | 98 ++
.../org/apache/pulsar/admin/cli/package-info.java | 24 +
.../org/apache/pulsar/client/admin/Functions.java | 143 +++
.../pulsar/client/admin/PulsarFunctionsAdmin.java | 65 ++
.../client/admin/internal/FunctionsImpl.java | 140 +++
.../apache/pulsar/admin/cli/CmdFunctionsTest.java | 397 ++++++++
pulsar-functions/cli/src/test/resources/log4j2.yml | 37 +
.../src/test/resources/test_function_config.yml | 24 +
pulsar-functions/conf/client.conf | 27 +
pulsar-functions/conf/example.yml | 30 +
pulsar-functions/conf/example_logging.yml | 27 +
pulsar-functions/conf/function_worker.yml | 46 +
pulsar-functions/conf/log4j2.yml | 167 ++++
{conf => pulsar-functions/conf}/pulsar_env.sh | 0
.../conf}/pulsar_tools_env.sh | 0
pulsar-functions/dist/pom.xml | 109 +++
pulsar-functions/dist/src/assemble/bin.xml | 83 ++
.../dist}/src/assemble/src.xml | 0
pulsar-functions/instance/pom.xml | 227 +++++
.../pulsar/functions/instance/ContextImpl.java | 272 ++++++
.../functions/instance/FunctionResultRouter.java | 56 ++
.../pulsar/functions/instance/FunctionStats.java | 143 +++
.../pulsar/functions/instance/InstanceConfig.java | 43 +
.../functions/instance/JavaExecutionResult.java | 43 +
.../pulsar/functions/instance/JavaInstance.java | 95 ++
.../functions/instance/JavaInstanceRunnable.java | 764 +++++++++++++++
.../pulsar/functions/instance/package-info.java | 22 +
.../producers/AbstractOneSinkTopicProducers.java | 72 ++
.../MultiConsumersOneSinkTopicProducers.java | 103 ++
.../functions/instance/producers/Producers.java | 59 ++
.../producers/SimpleOneSinkTopicProducers.java | 63 ++
.../functions/instance/state/StateContext.java | 32 +
.../functions/instance/state/StateContextImpl.java | 62 ++
.../instance/src/main/python/Function_pb2.py | 545 +++++++++++
.../src/main/python/InstanceCommunication_pb2.py | 505 ++++++++++
.../main/python/InstanceCommunication_pb2_grpc.py | 86 ++
.../instance/src/main/python/__init__.py | 21 +
.../instance/src/main/python/contextimpl.py | 160 ++++
pulsar-functions/instance/src/main/python/log.py | 111 +++
.../instance/src/main/python/python_instance.py | 310 ++++++
.../src/main/python/python_instance_main.py | 149 +++
.../instance/src/main/python/server.py | 70 ++
pulsar-functions/instance/src/main/python/util.py | 83 ++
.../instance/FunctionResultRouterTest.java | 117 +++
.../instance/JavaInstanceRunnableProcessTest.java | 1006 +++++++++++++++++++
.../instance/JavaInstanceRunnableTest.java | 234 +++++
.../functions/instance/JavaInstanceTest.java | 59 ++
.../MultiConsumersOneSinkTopicProducersTest.java | 124 +++
.../producers/SimpleOneSinkTopicProducersTest.java | 94 ++
pulsar-functions/java-examples/pom.xml | 46 +
.../functions/api/examples/CounterFunction.java | 35 +
.../api/examples/ExclamationFunction.java | 30 +
.../api/examples/JavaNativeExclmationFunction.java | 28 +
.../functions/api/examples/LoggingFunction.java | 45 +
.../functions/api/examples/PublishFunction.java | 31 +
.../functions/api/examples/UserConfigFunction.java | 30 +
.../functions/api/examples/UserMetricFunction.java | 31 +
.../functions/api/examples/VoidFunction.java | 29 +
pulsar-functions/metrics/pom.xml | 106 ++
.../pulsar/functions/metrics/MetricsSink.java | 63 ++
.../pulsar/functions/metrics/package-info.java | 22 +
.../functions/metrics/sink/AbstractWebSink.java | 139 +++
.../pulsar/functions/metrics/sink/FileSink.java | 127 +++
.../functions/metrics/sink/PrometheusSink.java | 182 ++++
.../functions/metrics/sink/FileSinkTest.java | 116 +++
.../metrics/sink/PrometheusSinkTests.java | 139 +++
pulsar-functions/pom.xml | 67 ++
pulsar-functions/proto/pom.xml | 82 ++
.../proto/src/main/proto/Function.proto | 81 ++
.../src/main/proto/InstanceCommunication.proto | 71 ++
.../proto/src/main/proto/Request.proto | 44 +
pulsar-functions/python-examples/exclamation.py | 29 +
.../pure_python_function_exclamation.py | 22 +
pulsar-functions/python-examples/thumbnailer.py | 29 +
pulsar-functions/run-counter-examples.sh | 26 +
pulsar-functions/run-examples.sh | 25 +
pulsar-functions/run-logging-examples.sh | 25 +
pulsar-functions/run-publish-example.sh | 23 +
pulsar-functions/runtime/pom.xml | 276 ++++++
.../org/apache/pulsar/functions/package-info.java | 23 +
.../pulsar/functions/runtime/JavaInstanceMain.java | 219 +++++
.../pulsar/functions/runtime/ProcessRuntime.java | 307 ++++++
.../functions/runtime/ProcessRuntimeFactory.java | 108 +++
.../apache/pulsar/functions/runtime/Runtime.java | 45 +
.../pulsar/functions/runtime/RuntimeFactory.java | 41 +
.../pulsar/functions/runtime/RuntimeSpawner.java | 130 +++
.../pulsar/functions/runtime/ThreadRuntime.java | 146 +++
.../functions/runtime/ThreadRuntimeFactory.java | 89 ++
.../pulsar/functions/runtime/package-info.java | 23 +
.../src/main/resources/java_instance_log4j2.yml | 167 ++++
.../functions/runtime/ProcessRuntimeTest.java | 150 +++
.../runtime/src/test/resources/log4j2.yml | 37 +
pulsar-functions/scripts/python/README.txt | 2 +
pulsar-functions/scripts/python/generate.sh | 40 +
pulsar-functions/scripts/python/requirements.txt | 0
pulsar-functions/scripts/python/setup.py.template | 44 +
pulsar-functions/submit-python-function.sh | 28 +
pulsar-functions/utils/pom.xml | 182 ++++
.../apache/pulsar/functions/utils/Exceptions.java | 51 +
.../functions/utils/FunctionConfigUtils.java | 98 ++
.../apache/pulsar/functions/utils/Reflections.java | 206 ++++
.../org/apache/pulsar/functions/utils/Utils.java | 52 +
.../utils/functioncache/FunctionCacheEntry.java | 106 ++
.../utils/functioncache/FunctionCacheManager.java | 80 ++
.../functioncache/FunctionCacheManagerImpl.java | 134 +++
.../utils/functioncache/FunctionClassLoaders.java | 49 +
.../utils/functioncache/package-info.java | 23 +
.../pulsar/functions/utils/package-info.java | 23 +
.../pulsar/functions/utils/ExceptionsTest.java | 80 ++
.../pulsar/functions/utils/ReflectionsTest.java | 167 ++++
.../apache/pulsar/functions/utils/UtilsTest.java | 55 ++
.../functioncache/FunctionCacheEntryTest.java | 146 +++
.../FunctionCacheManagerImplTest.java | 161 ++++
.../functioncache/FunctionClassLoadersTest.java | 56 ++
.../utils/src/test/resources/multifunction.jar | Bin 0 -> 1744 bytes
pulsar-functions/worker/pom.xml | 131 +++
.../worker/ClusterServiceCoordinator.java | 87 ++
.../pulsar/functions/worker/FunctionAction.java | 39 +
.../pulsar/functions/worker/FunctionActioner.java | 193 ++++
.../functions/worker/FunctionAssignmentTailer.java | 102 ++
.../functions/worker/FunctionMetaDataManager.java | 415 ++++++++
.../worker/FunctionMetaDataTopicTailer.java | 97 ++
.../functions/worker/FunctionMetadataSetup.java | 126 +++
.../functions/worker/FunctionRuntimeInfo.java | 38 +
.../functions/worker/FunctionRuntimeManager.java | 559 +++++++++++
.../functions/worker/FunctionWorkerStarter.java | 67 ++
.../pulsar/functions/worker/MembershipManager.java | 289 ++++++
.../pulsar/functions/worker/SchedulerManager.java | 227 +++++
.../org/apache/pulsar/functions/worker/Utils.java | 209 ++++
.../org/apache/pulsar/functions/worker/Worker.java | 80 ++
.../pulsar/functions/worker/WorkerConfig.java | 125 +++
.../pulsar/functions/worker/WorkerService.java | 187 ++++
.../functions/worker/dlog/DLInputStream.java | 145 +++
.../functions/worker/dlog/DLOutputStream.java | 73 ++
.../functions/worker/request/RequestResult.java | 40 +
.../worker/request/ServiceRequestInfo.java | 52 +
.../worker/request/ServiceRequestManager.java | 53 +
.../worker/request/ServiceRequestUtils.java | 61 ++
.../worker/rest/ConfigurationResource.java | 48 +
.../functions/worker/rest/FunctionApiResource.java | 41 +
.../pulsar/functions/worker/rest/Resources.java | 45 +
.../pulsar/functions/worker/rest/RestUtils.java | 39 +
.../pulsar/functions/worker/rest/WorkerServer.java | 73 ++
.../worker/rest/api/v1/FunctionApiV1Resource.java | 542 +++++++++++
.../functions/worker/scheduler/IScheduler.java | 30 +
.../worker/scheduler/RoundRobinScheduler.java | 73 ++
.../worker/ClusterServiceCoordinatorTest.java | 112 +++
.../worker/FunctionMetaDataManagerTest.java | 443 +++++++++
.../worker/FunctionMetaDataTopicTailerTest.java | 84 ++
.../worker/FunctionRuntimeManagerTest.java | 389 ++++++++
.../functions/worker/MembershipManagerTest.java | 314 ++++++
.../functions/worker/SchedulerManagerTest.java | 703 ++++++++++++++
.../pulsar/functions/worker/WorkerConfigTest.java | 63 ++
.../functions/worker/dlog/DLInputStreamTest.java | 105 ++
.../functions/worker/dlog/DLOutputStreamTest.java | 77 ++
.../functions/worker/executor/MockClock.java | 65 ++
.../functions/worker/executor/MockClockTest.java | 46 +
.../worker/executor/MockExecutorController.java | 215 +++++
.../executor/MockExecutorControllerTest.java | 104 ++
.../worker/request/ServiceRequestManagerTest.java | 70 ++
.../rest/api/v1/FunctionApiV1ResourceTest.java | 1008 ++++++++++++++++++++
.../worker/src/test/resources/log4j2.yml | 37 +
.../src/test/resources/test_worker_config.yml | 25 +
181 files changed, 22548 insertions(+)
create mode 100644 pulsar-functions/api-java/pom.xml
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/Context.java
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/PulsarFunction.java
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/SerDe.java
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/package-info.java
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/utils/DefaultSerDe.java
create mode 100644 pulsar-functions/api-java/src/main/java/org/apache/pulsar/functions/api/utils/JavaSerDe.java
create mode 100644 pulsar-functions/api-java/src/test/java/org/apache/pulsar/functions/api/utils/DefaultSerDeTest.java
create mode 100644 pulsar-functions/api-java/src/test/java/org/apache/pulsar/functions/api/utils/JavaSerDeTest.java
create mode 100644 pulsar-functions/api-python/__init__.py
create mode 100644 pulsar-functions/api-python/context.py
create mode 100644 pulsar-functions/api-python/pulsar_function.py
create mode 100644 pulsar-functions/api-python/serde.py
create mode 100755 pulsar-functions/bin/pulsar
create mode 100755 pulsar-functions/bin/pulsar-functions
create mode 100644 pulsar-functions/cli/pom.xml
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/admin/cli/CmdFunctions.java
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/admin/cli/FunctionsTool.java
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/admin/cli/package-info.java
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/client/admin/Functions.java
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/client/admin/PulsarFunctionsAdmin.java
create mode 100644 pulsar-functions/cli/src/main/java/org/apache/pulsar/client/admin/internal/FunctionsImpl.java
create mode 100644 pulsar-functions/cli/src/test/java/org/apache/pulsar/admin/cli/CmdFunctionsTest.java
create mode 100644 pulsar-functions/cli/src/test/resources/log4j2.yml
create mode 100644 pulsar-functions/cli/src/test/resources/test_function_config.yml
create mode 100644 pulsar-functions/conf/client.conf
create mode 100644 pulsar-functions/conf/example.yml
create mode 100644 pulsar-functions/conf/example_logging.yml
create mode 100644 pulsar-functions/conf/function_worker.yml
create mode 100644 pulsar-functions/conf/log4j2.yml
copy {conf => pulsar-functions/conf}/pulsar_env.sh (100%)
copy {conf => pulsar-functions/conf}/pulsar_tools_env.sh (100%)
create mode 100644 pulsar-functions/dist/pom.xml
create mode 100644 pulsar-functions/dist/src/assemble/bin.xml
copy {all => pulsar-functions/dist}/src/assemble/src.xml (100%)
create mode 100644 pulsar-functions/instance/pom.xml
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/ContextImpl.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/FunctionResultRouter.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/FunctionStats.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/InstanceConfig.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaExecutionResult.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstance.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/package-info.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/producers/AbstractOneSinkTopicProducers.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/producers/MultiConsumersOneSinkTopicProducers.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/producers/Producers.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/producers/SimpleOneSinkTopicProducers.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContext.java
create mode 100644 pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/state/StateContextImpl.java
create mode 100644 pulsar-functions/instance/src/main/python/Function_pb2.py
create mode 100644 pulsar-functions/instance/src/main/python/InstanceCommunication_pb2.py
create mode 100644 pulsar-functions/instance/src/main/python/InstanceCommunication_pb2_grpc.py
create mode 100644 pulsar-functions/instance/src/main/python/__init__.py
create mode 100644 pulsar-functions/instance/src/main/python/contextimpl.py
create mode 100644 pulsar-functions/instance/src/main/python/log.py
create mode 100644 pulsar-functions/instance/src/main/python/python_instance.py
create mode 100644 pulsar-functions/instance/src/main/python/python_instance_main.py
create mode 100644 pulsar-functions/instance/src/main/python/server.py
create mode 100644 pulsar-functions/instance/src/main/python/util.py
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/FunctionResultRouterTest.java
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceRunnableProcessTest.java
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceRunnableTest.java
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/JavaInstanceTest.java
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/producers/MultiConsumersOneSinkTopicProducersTest.java
create mode 100644 pulsar-functions/instance/src/test/java/org/apache/pulsar/functions/instance/producers/SimpleOneSinkTopicProducersTest.java
create mode 100644 pulsar-functions/java-examples/pom.xml
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/CounterFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/ExclamationFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/JavaNativeExclmationFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/LoggingFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/PublishFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/UserConfigFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/UserMetricFunction.java
create mode 100644 pulsar-functions/java-examples/src/main/java/org/apache/pulsar/functions/api/examples/VoidFunction.java
create mode 100644 pulsar-functions/metrics/pom.xml
create mode 100644 pulsar-functions/metrics/src/main/java/org/apache/pulsar/functions/metrics/MetricsSink.java
create mode 100644 pulsar-functions/metrics/src/main/java/org/apache/pulsar/functions/metrics/package-info.java
create mode 100644 pulsar-functions/metrics/src/main/java/org/apache/pulsar/functions/metrics/sink/AbstractWebSink.java
create mode 100644 pulsar-functions/metrics/src/main/java/org/apache/pulsar/functions/metrics/sink/FileSink.java
create mode 100644 pulsar-functions/metrics/src/main/java/org/apache/pulsar/functions/metrics/sink/PrometheusSink.java
create mode 100644 pulsar-functions/metrics/src/test/java/org/apache/pulsar/functions/metrics/sink/FileSinkTest.java
create mode 100644 pulsar-functions/metrics/src/test/java/org/apache/pulsar/functions/metrics/sink/PrometheusSinkTests.java
create mode 100644 pulsar-functions/pom.xml
create mode 100644 pulsar-functions/proto/pom.xml
create mode 100644 pulsar-functions/proto/src/main/proto/Function.proto
create mode 100644 pulsar-functions/proto/src/main/proto/InstanceCommunication.proto
create mode 100644 pulsar-functions/proto/src/main/proto/Request.proto
create mode 100755 pulsar-functions/python-examples/exclamation.py
create mode 100755 pulsar-functions/python-examples/pure_python_function_exclamation.py
create mode 100755 pulsar-functions/python-examples/thumbnailer.py
create mode 100755 pulsar-functions/run-counter-examples.sh
create mode 100755 pulsar-functions/run-examples.sh
create mode 100755 pulsar-functions/run-logging-examples.sh
create mode 100755 pulsar-functions/run-publish-example.sh
create mode 100644 pulsar-functions/runtime/pom.xml
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/package-info.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/JavaInstanceMain.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntime.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ProcessRuntimeFactory.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/Runtime.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeFactory.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeSpawner.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntime.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/ThreadRuntimeFactory.java
create mode 100644 pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/package-info.java
create mode 100644 pulsar-functions/runtime/src/main/resources/java_instance_log4j2.yml
create mode 100644 pulsar-functions/runtime/src/test/java/org/apache/pulsar/functions/runtime/ProcessRuntimeTest.java
create mode 100644 pulsar-functions/runtime/src/test/resources/log4j2.yml
create mode 100644 pulsar-functions/scripts/python/README.txt
create mode 100644 pulsar-functions/scripts/python/generate.sh
create mode 100644 pulsar-functions/scripts/python/requirements.txt
create mode 100644 pulsar-functions/scripts/python/setup.py.template
create mode 100755 pulsar-functions/submit-python-function.sh
create mode 100644 pulsar-functions/utils/pom.xml
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/Exceptions.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/FunctionConfigUtils.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/Reflections.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/Utils.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/functioncache/FunctionCacheEntry.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/functioncache/FunctionCacheManager.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/functioncache/FunctionCacheManagerImpl.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/functioncache/FunctionClassLoaders.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/functioncache/package-info.java
create mode 100644 pulsar-functions/utils/src/main/java/org/apache/pulsar/functions/utils/package-info.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/ExceptionsTest.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/ReflectionsTest.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/UtilsTest.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/functioncache/FunctionCacheEntryTest.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/functioncache/FunctionCacheManagerImplTest.java
create mode 100644 pulsar-functions/utils/src/test/java/org/apache/pulsar/functions/utils/functioncache/FunctionClassLoadersTest.java
create mode 100644 pulsar-functions/utils/src/test/resources/multifunction.jar
create mode 100644 pulsar-functions/worker/pom.xml
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinator.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionAction.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionActioner.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionAssignmentTailer.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataManager.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetaDataTopicTailer.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionMetadataSetup.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeInfo.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionRuntimeManager.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/FunctionWorkerStarter.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/MembershipManager.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Utils.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/Worker.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerConfig.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/WorkerService.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLInputStream.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/dlog/DLOutputStream.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/request/RequestResult.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/request/ServiceRequestInfo.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/request/ServiceRequestManager.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/request/ServiceRequestUtils.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/ConfigurationResource.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/FunctionApiResource.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/Resources.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/RestUtils.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/WorkerServer.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/v1/FunctionApiV1Resource.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/scheduler/IScheduler.java
create mode 100644 pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/scheduler/RoundRobinScheduler.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/ClusterServiceCoordinatorTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/FunctionMetaDataManagerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/FunctionMetaDataTopicTailerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/FunctionRuntimeManagerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/MembershipManagerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/SchedulerManagerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/WorkerConfigTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLInputStreamTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/dlog/DLOutputStreamTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/executor/MockClock.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/executor/MockClockTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/executor/MockExecutorController.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/executor/MockExecutorControllerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/request/ServiceRequestManagerTest.java
create mode 100644 pulsar-functions/worker/src/test/java/org/apache/pulsar/functions/worker/rest/api/v1/FunctionApiV1ResourceTest.java
create mode 100644 pulsar-functions/worker/src/test/resources/log4j2.yml
create mode 100644 pulsar-functions/worker/src/test/resources/test_worker_config.yml
--
To stop receiving notification emails like this one, please contact
mmerli@apache.org.