You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by HeartSaVioR <gi...@git.apache.org> on 2016/09/19 09:24:51 UTC

[GitHub] storm pull request #1689: STORM-2099 Introduce new sql external module: stor...

GitHub user HeartSaVioR opened a pull request:

    https://github.com/apache/storm/pull/1689

    STORM-2099 Introduce new sql external module: storm-sql-redis

    This patch is on top of [STORM-2089](https://issues.apache.org/jira/browse/STORM-2089) (#1682)
    
    I moved JsonScheme and JsonSerializer to storm-sql-runtime since it's used from storm-sql-kafka and storm-sql-redis. In result, additional dependency (jackson-databind) for storm-sql-runtime is introduced.
    
    Below is the example of defining Redis table:
    
    ```
    CREATE EXTERNAL TABLE LARGE_ORDERS (ID INT PRIMARY KEY, TOTAL INT) LOCATION 'redis://localhost:7379' TBLPROPERTIES '{"data.type": "STRING", "data.additional.key": "addkey", "redis.timeout": "3000", "use.redis.cluster": "true"}'
    ```
    
    LOCATION can be set to `redis://:[password]@[host]:[port]/[dbIdx]`.
    Default mode is JedisPool (for single Redis). If you want to enable cluster mode you need to specify option "use.redis.cluster" to "true".
    
    In cluster mode password and dbIdx is ignored. DB index is not valid for Cluster mode, but recently password for Cluster mode of Redis becomes valid. 
    This will be addressed from storm-redis and addressed to storm-sql-redis later.
    
    Please note that Redis(Cluster)State only supports STRING and HASH since State stores key and value.
    
    Please review and comment. Thanks in advance!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/HeartSaVioR/storm STORM-2099

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/storm/pull/1689.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1689
    
----
commit 3c29427ad9af26e2cb4519848ede96a1e2848669
Author: Jungtaek Lim <ka...@gmail.com>
Date:   2016-09-13T02:40:13Z

    STORM-2089 Replace Consumer of ISqlTridentDataSource with SqlTridentConsumer
    
    * SqlTridentConsumer contains StateFactory and StateUpdater which is needed to store tuples to State via batch
    * Apply the change to storm-sql-kafka
    * add javadoc to ISqlTridentDataSource

commit 7f0fffc5a92d148d01318cf591e4916f153ea077
Author: Jungtaek Lim <ka...@gmail.com>
Date:   2016-09-19T08:58:54Z

    STORM-2099 Introduce new sql external module: storm-sql-redis
    
    * implements RedisDataSourcesProvider
      * Redis data source only provides Consumer (State)
    * move out JsonScheme and JsonSerializer to runtime
      * it's both used for storm-sql-kafka and storm-sql-redis

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1689: STORM-2099 Introduce new sql external module: storm-sql-r...

Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the issue:

    https://github.com/apache/storm/pull/1689
  
    +1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1689: STORM-2099 Introduce new sql external module: storm-sql-r...

Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:

    https://github.com/apache/storm/pull/1689
  
    Rebased to reflect merged STORM-2089.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm pull request #1689: STORM-2099 Introduce new sql external module: stor...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/storm/pull/1689


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1689: STORM-2099 Introduce new sql external module: storm-sql-r...

Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:

    https://github.com/apache/storm/pull/1689
  
    There're some compile issues from test side. I'll handle it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1689: STORM-2099 Introduce new sql external module: storm-sql-r...

Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:

    https://github.com/apache/storm/pull/1689
  
    Build for '!storm-core' passed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] storm issue #1689: STORM-2099 Introduce new sql external module: storm-sql-r...

Posted by HeartSaVioR <gi...@git.apache.org>.
Github user HeartSaVioR commented on the issue:

    https://github.com/apache/storm/pull/1689
  
    In order to add Redis as data source for SQL, I added two artifacts to --artifacts parameter.
    
    - org.apache.storm:storm-sql-redis:2.0.0-SNAPSHOT
    - org.apache.storm:storm-redis:2.0.0-SNAPSHOT
    
    Might be better to have a separate document for explaining the data sources for SQL. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---