You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by traflm <gi...@git.apache.org> on 2018/08/25 15:26:36 UTC
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
GitHub user traflm opened a pull request:
https://github.com/apache/trafodion/pull/1704
[TRAFODION-2952] large amount of data will cause error in sequence ge…
…nerating
Currently, the sequence next range value is NOT protected by DTM, but a retry logic.
If multiple processes try to use the same sequence, when the cache is used, all of them try to get the next range and go into race. So one may get a range that another get. The logic is any one get the next range will check the updated timestamp, if it is itself, then it is good for him. Otherwise, the range is got by someone else. So the code wait and retry.
For the use case described in JIRA TRAFODION-2952, there are 48 partitioned table and want to do a load of 4 billion rows. It is high concurrency, and many times conflict.
The that large load, it is very frequent to fail, since the retry time is hardcoded to 10.
This patch introduce a CQD to control the retry number. It is set to a higher number by default, so in most cases, users don't need to modify this. But if still run into issue, one can change the CQD.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/traflm/trafodion TRAFODION-2952
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafodion/pull/1704.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1704
----
commit 8958ce6aa28a96a15ed067dc49f556e0945165a6
Author: Liu Ming <ov...@...>
Date: 2018-08-25T15:20:32Z
[TRAFODION-2952] large amount of data will cause error in sequence generating
----
---
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
Posted by traflm <gi...@git.apache.org>.
Github user traflm commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1704#discussion_r213515995
--- Diff: core/sql/cli/Cli.cpp ---
@@ -10599,7 +10599,10 @@ static Lng32 SeqGenCliInterfaceUpdAndValidateMulti(
numTries++;
- DELAY(100 + numTries*25);
+ if( 100 + numTries*25 < 1000) //MAX is 1 second
+ DELAY(100 + numTries*25);
+ else
+ DELAY(1000);
--- End diff --
Yes, this is a good suggestion
---
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1704#discussion_r213058744
--- Diff: core/sql/cli/Cli.cpp ---
@@ -10599,7 +10599,10 @@ static Lng32 SeqGenCliInterfaceUpdAndValidateMulti(
numTries++;
- DELAY(100 + numTries*25);
+ if( 100 + numTries*25 < 1000) //MAX is 1 second
+ DELAY(100 + numTries*25);
+ else
+ DELAY(1000);
--- End diff --
You might want to consider adding some small random amount to this maximum delay. Otherwise you might get a bunch of processes in lock step, all retrying at the same time.
---
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1704#discussion_r213058185
--- Diff: core/sql/sqlcomp/DefaultConstants.h ---
@@ -2946,6 +2946,12 @@ enum DefaultConstants
// this is used to change cache size of sequence numbers for a session.
// It overwrites the cache size that was specified during sequence creation.
TRAF_SEQUENCE_CACHE_SIZE,
+
+ // this is used to set the retry time if two concurrent update of sequence
+ // conflict, and how many times will retry
+ // by default it is 250, when you saw error 1583, you can try to increase
--- End diff --
In nadefaults.cpp, it looks like the default is 100, not 250, so the comment is wrong.
---
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
Posted by traflm <gi...@git.apache.org>.
Github user traflm commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1704#discussion_r213515950
--- Diff: core/sql/sqlcomp/DefaultConstants.h ---
@@ -2946,6 +2946,12 @@ enum DefaultConstants
// this is used to change cache size of sequence numbers for a session.
// It overwrites the cache size that was specified during sequence creation.
TRAF_SEQUENCE_CACHE_SIZE,
+
+ // this is used to set the retry time if two concurrent update of sequence
+ // conflict, and how many times will retry
+ // by default it is 250, when you saw error 1583, you can try to increase
--- End diff --
thanks Dave, yes, I will change this
---
[GitHub] trafodion pull request #1704: [TRAFODION-2952] large amount of data will cau...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/trafodion/pull/1704
---