You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Parth-Brahmbhatt <gi...@git.apache.org> on 2015/01/06 04:32:00 UTC
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
GitHub user Parth-Brahmbhatt opened a pull request:
https://github.com/apache/storm/pull/372
Storm 616 : Storm-jdbc connector.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Parth-Brahmbhatt/incubator-storm STORM-616
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/372.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #372
----
commit 65e9f0c814b2cddc772880042259b66194fd6fb7
Author: Parth Brahmbhatt <br...@gmail.com>
Date: 2014-12-05T22:48:34Z
STORM-586: TridentKafkaEmitter should catch updateOffsetException.
commit 5b160168c75c0e8c4c402a5e24f606dab697fbef
Author: Parth Brahmbhatt <br...@gmail.com>
Date: 2015-01-06T03:14:18Z
STORM-616: Jdbc connector for storm.
commit ab9f778ae50a1e224ebdcc58e6249009fc1f91cc
Author: Parth Brahmbhatt <br...@gmail.com>
Date: 2015-01-06T03:23:52Z
Merge remote-tracking branch 'upstream/master' into STORM-616
commit d260759ac203383e27668a7cb7090926029f7406
Author: Parth Brahmbhatt <br...@gmail.com>
Date: 2015-01-06T03:31:05Z
STORM-616 : removing unintended changes.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt commented on the pull request:
https://github.com/apache/storm/pull/372#issuecomment-68904233
* I noticed that and reverted those files to their state in master.
* Added storm-jdbc to storm-core.
* sorry about this, I accidently deleted it with mysql dependency that i added for testing. I have added it back.
* jdbc.conf is just a key in storm config that holds a map for hikariCP configuration. I have added a link to HikariCP configuration document.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/372#issuecomment-68871971
@Parth-Brahmbhatt looks like you might have unintended commit as part of this PR.
"STORM-586: TridentKafkaEmitter should catch updateOffsetException."
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on the pull request:
https://github.com/apache/storm/pull/372#issuecomment-68899213
@Parth-Brahmbhatt few things to note
1) add external/storm-jdbc to storm/pom.xml under modules
2) I didn't' find any dependency in storm-jdbc/pom.xml for com.google.common and its causing compilation errors package com.google.common.base does not exist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
Posted by Parth-Brahmbhatt <gi...@git.apache.org>.
Github user Parth-Brahmbhatt closed the pull request at:
https://github.com/apache/storm/pull/372
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] storm pull request: Storm 616 : Storm-jdbc connector.
Posted by harshach <gi...@git.apache.org>.
Github user harshach commented on a diff in the pull request:
https://github.com/apache/storm/pull/372#discussion_r22536766
--- Diff: external/storm-jdbc/README.md ---
@@ -0,0 +1,117 @@
+#Storm HBase
+
+Storm/Trident integration for JDBC.
+
+## Usage
+The main API for interacting with JDBC is the `org.apache.storm.jdbc.mapper.TupleToColumnMapper`
+interface:
+
+```java
+public interface JdbcMapper extends Serializable {
+ List<Column> getColumns(ITuple tuple);
+}
+```
+
+The `getColumns()` method defines how a storm tuple maps to a list of columns representing a row in a database.
+
+### SimpleJdbcMapper
+`storm-jdbc` includes a general purpose `JdbcMapper` implementation called `SimpleJdbcMapper` that can map Storm
+tuple to a Database row. `SimpleJdbcMapper` assumes that the tuple has fields with same name as the column name in
+the database table that you intend to write to.
+
+To use `SimpleJdbcMapper`, you simply tell it the tableName that you want to write to and provide a hikari configuration map.
+
+The following code creates a `SimpleJdbcMapper` instance that:
+
+1. Will allow the mapper to transform a storm tuple to a list of columns mapping to a row in table test.user_details.
+2. Will use the provided HikariCP configuration to establish a connection pool with specified Database configuration and
+automatically figure out the column names of the table that you intend to write to.
+
+```java
+Map hikariConfigMap = Maps.newHashMap();
+hikariConfigMap.put("dataSourceClassName","com.mysql.jdbc.jdbc2.optional.MysqlDataSource");
+hikariConfigMap.put("dataSource.url", "jdbc:mysql://localhost/test");
+hikariConfigMap.put("dataSource.user","root");
+hikariConfigMap.put("dataSource.password","password");
+String tableName = "user_details";
+JdbcMapper jdbcMapper = new SimpleJdbcMapper(tableName, map);
+```
+### JdbcBolt
+To use the `JdbcBolt`, construct it with the name of the table to write to, and a `JdbcMapper` implementation. In addition
+you must specify a configuration key that hold the hikari configuration map.
+
+ ```java
+Config config = new Config();
+config.put("jdbc.conf", hikariConfigMap);
+
+JdbcBolt bolt = new JdbcBolt("user_details", jdbcMapper)
+ .withConfigKey("jdbc.conf");
--- End diff --
is jdbc.conf is a properties file? if so withConfigKey can be renamed to withConfigFile. It will be great if you can add a sample jdbc.conf file to the README.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---