You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by agsachin <gi...@git.apache.org> on 2016/04/20 07:33:43 UTC
[GitHub] spark pull request: [SPARK-14745] [Streaming] [WIP] CEP support in...
GitHub user agsachin opened a pull request:
https://github.com/apache/spark/pull/12518
[SPARK-14745] [Streaming] [WIP] CEP support in Spark Streaming
## What changes were proposed in this pull request?
This JIRA is about how/what can we add in Spark Streaming to support CEP out of the box.
The code in this PR needs to merge up with Structured Streaming changes in 2.0. The PR is issued for now to facilitate discussion and view by other users interested in CEP and asked on spark-user group
## How was this patch tested?
Manually with KafkaDirect. WIP
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/agsachin/spark CEP
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12518.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12518
----
commit 87032d23a1ebfa909408edb193763165f351383a
Author: sachin aggarwal <di...@gmail.com>
Date: 2016-03-23T11:57:53Z
adding initial code for CEP
commit cca266e84f123061d186ad8d9f8fdaafc6bff236
Author: sachin aggarwal <di...@gmail.com>
Date: 2016-03-23T12:00:54Z
adding initial code for CEP
commit 633408de388db4decc38616327e588477e7147d8
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-30T14:51:38Z
sortedPrevMappedStream created incrementally
commit 087bac9bf8158ff1b556166c24a039fffc5b9f3d
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-31T13:03:57Z
add back dynamic window
commit 750c9716626b1bf0213625cc7b4365399c063d81
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-31T14:03:14Z
remove println & api Cleanup
commit 9a4d32dcc01bce30a3322fac95be3ec839dbcd59
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-31T14:09:54Z
changed WindowMetric to WindowState
commit e433602529dfb5b84eaf0390b8324c40a6e2d7a7
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-31T14:15:41Z
undo add to test suite
commit 5e67477f5dd187a371f1f6916fae2e75a652f4ac
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-03-31T14:17:47Z
more undo to test suite
commit 94410a75a279261e62a0bdd5aaf9bda74c08a022
Author: Mario Briggs <ma...@in.ibm.com>
Date: 2016-04-13T17:28:15Z
Per part commit
commit b16f1389f5c64964e12bb5d29b5f2ce09428cc61
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-04-19T18:34:24Z
Support on PairDStream to enable pattern matching by key
commit 8e5a5b28b3872f4f6620eb5bbf86aab84d83bfcd
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-04-19T18:43:21Z
missed file KeyedPatternMatchDStrean
commit 162ffbe39e41e958bbf31bc68e0f5cbe2d68ec78
Author: mariobriggs <ma...@in.ibm.com>
Date: 2016-04-20T04:41:19Z
CEP samples using DirectKafkaInputDStream
commit 1e1c6703c7b6dfa17b88c171029d9b689c584244
Author: sachin aggarwal <di...@gmail.com>
Date: 2016-04-20T04:41:48Z
Merge pull request #2 from agsachin/cep-marioPerPar
Support for Pattern match on PairDStream
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Spark Str...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/12518
Hello @agsachin,
This is a cool idea, and we apologize that this has been sitting here for a while. Unfortunately, add-on features like this is unlikely to be core API for DStreams/DataFrames. So this is something that is best contributed to open-source as a [spark-package](spark-packages.org).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Spark Str...
Posted by dpinol <gi...@git.apache.org>.
Github user dpinol commented on the issue:
https://github.com/apache/spark/pull/12518
@agsachin Hi, do you plan to create a spark-package with this?
thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Spark Str...
Posted by opsnull <gi...@git.apache.org>.
Github user opsnull commented on the issue:
https://github.com/apache/spark/pull/12518
@rxin HI，do you have any plan on CEP ？
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-14745] [Streaming] [WIP] CEP support in...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/12518#issuecomment-212267044
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Sp...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/12518
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Spark Str...
Posted by tdas <gi...@git.apache.org>.
Github user tdas commented on the issue:
https://github.com/apache/spark/pull/12518
@rxin thanks for the clarification!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #12518: [SPARK-14745] [Streaming] [WIP] CEP support in Spark Str...
Posted by rxin <gi...@git.apache.org>.
Github user rxin commented on the issue:
https://github.com/apache/spark/pull/12518
@tdas I wouldn't rule it out as a possibility in the future, but I agree currently the project needs to focus more on structured streaming APIs first and then think about more complicated use cases like CEP.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org