You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com> on 2017/03/23 05:47:37 UTC

performing update on db table using storm

For Eg : Think we are getting three json chucks of some post with increase
in comment count and new text.while first json chuck content will get a
succeeded insert , while second and third json chucks share the same post
id with the different comment count for update which are sent to db at same
time by using parallelism of storm (Two different update queries are trying
to perform at same time ).which is causing dead lock, I can see two
transactions holding same id in db for update.

I just wrote a insert query and below that I have written a update query in
single bolt on increasing parallelism to that bolt,I can see more dead
locks on db table.

Writing the a different update bolt will solve by problem of dead locks ?



*Best regards,*

*K.Sai Dilip Reddy.*

Re: performing update on db table using storm

Posted by Punit Tiwan <ti...@coviam.com>.
Hi Dilip,

Use "Field Grouping” will be one way of handling this. 

Instead of “Shuffle Grouping”, if you use “Field Grouping” and use “user_id” as field then a particular Json will always go to a particular instance of Bolt. Hence two update for the same “id” won’t happen.

Regards,
Punit

> On 24-Mar-2017, at 8:10 PM, Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com> wrote:
> 
> HI Navin,
> 
> Thanks,I'm using storm for ETL work where I get Json chucks. By using storm.jdbc I'm storing the required fields in postgresdb tables.I need to update some rows based on user name.
> 
> Eg: mapping required fields
> 
> java List<Column> columnSchema = Lists.newArrayList( new Column("user_id", java.sql.Types.INTEGER), new Column("user_name", java.sql.Types.VARCHAR)); JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(columnSchema); 
> 
> after insert query,I'm using update query and updating the table row fields in the same bolt.when I do like that with increasing in parallelism.I'm seeing more deadlocks on postgres db tables.Can you suggest me an idea on handling those dead locks (without not forming dead locks on execution of update).    
> 
>  
> 
> Best regards,
> K.Sai Dilip Reddy.
> 
> 
> On Fri, Mar 24, 2017 at 1:26 PM, Navin Ipe <navin.ipe@searchlighthealth.com <ma...@searchlighthealth.com>> wrote:
> To begin with, Dilip, could you please explain your problem to everyone a bit more clearly? It's hard to understand what you are communicating.
> 
> On Thu, Mar 23, 2017 at 11:17 AM, Sai Dilip Reddy Kiralam <dkiralam@aadhya-analytics.com <ma...@aadhya-analytics.com>> wrote:
> For Eg : Think we are getting three json chucks of some post with increase in comment count and new text.while first json chuck content will get a succeeded insert , while second and third json chucks share the same post id with the different comment count for update which are sent to db at same time by using parallelism of storm (Two different update queries are trying to perform at same time ).which is causing dead lock, I can see two transactions holding same id in db for update. 
> 
> I just wrote a insert query and below that I have written a update query in single bolt on increasing parallelism to that bolt,I can see more dead locks on db table.
> 
> Writing the a different update bolt will solve by problem of dead locks ? 
> 
> 
> 
> Best regards,
> K.Sai Dilip Reddy.
> 
> 
> 
> 
> -- 
> Regards,
> Navin
> 


Re: performing update on db table using storm

Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
HI Navin,

Thanks,I'm using storm for ETL work where I get Json chucks. By using
storm.jdbc I'm storing the required fields in postgresdb tables.I need to
update some rows based on user name.

Eg: mapping required fields

java List<Column> columnSchema = Lists.newArrayList( new Column("user_id",
java.sql.Types.INTEGER), new Column("user_name", java.sql.Types.VARCHAR));
JdbcMapper simpleJdbcMapper = new SimpleJdbcMapper(columnSchema);

after insert query,I'm using update query and updating the table row fields
in the same bolt.when I do like that with increasing in parallelism.I'm
seeing more deadlocks on postgres db tables.Can you suggest me an idea on
handling those dead locks (without not forming dead locks on execution of
update).



*Best regards,*

*K.Sai Dilip Reddy.*

On Fri, Mar 24, 2017 at 1:26 PM, Navin Ipe <na...@searchlighthealth.com>
wrote:

> To begin with, Dilip, could you please explain your problem to everyone a
> bit more clearly? It's hard to understand what you are communicating.
>
> On Thu, Mar 23, 2017 at 11:17 AM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> For Eg : Think we are getting three json chucks of some post with
>> increase in comment count and new text.while first json chuck content will
>> get a succeeded insert , while second and third json chucks share the same
>> post id with the different comment count for update which are sent to db at
>> same time by using parallelism of storm (Two different update queries are
>> trying to perform at same time ).which is causing dead lock, I can see two
>> transactions holding same id in db for update.
>>
>> I just wrote a insert query and below that I have written a update
>> query in single bolt on increasing parallelism to that bolt,I can see more
>> dead locks on db table.
>>
>> Writing the a different update bolt will solve by problem of dead locks ?
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>
>
>
> --
> Regards,
> Navin
>

Re: performing update on db table using storm

Posted by Navin Ipe <na...@searchlighthealth.com>.
To begin with, Dilip, could you please explain your problem to everyone a
bit more clearly? It's hard to understand what you are communicating.

On Thu, Mar 23, 2017 at 11:17 AM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:

> For Eg : Think we are getting three json chucks of some post with increase
> in comment count and new text.while first json chuck content will get a
> succeeded insert , while second and third json chucks share the same post
> id with the different comment count for update which are sent to db at same
> time by using parallelism of storm (Two different update queries are trying
> to perform at same time ).which is causing dead lock, I can see two
> transactions holding same id in db for update.
>
> I just wrote a insert query and below that I have written a update
> query in single bolt on increasing parallelism to that bolt,I can see more
> dead locks on db table.
>
> Writing the a different update bolt will solve by problem of dead locks ?
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>



-- 
Regards,
Navin