You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by sandhyasun <gi...@git.apache.org> on 2018/04/02 17:43:57 UTC
[GitHub] trafodion pull request #1503: [TRAFOODION-2949] Performance changes for LOB ...
GitHub user sandhyasun opened a pull request:
https://github.com/apache/trafodion/pull/1503
[TRAFOODION-2949] Performance changes for LOB and increasing limit for larger varchars/…
…bobs to use with batch input.
For LOB internal descriptor files, there was a check in compiler that prevented caching of NA Table . Removed that restriction for LOB descriptor tables which affected evey DML into a table with a LOB column.
Fix to allow larger varchar/lob data to be input with batch input.
1. First check for rowset size is in pre codegen phase. This was added during POC since we were getting 4 GB of rowset data all at once.
CQD MEMORY_LIMIT_ROWSET_IN_MB was added and set to 500MB. The data + rowset size needed to fit into this space. I have changed it to be 1GB by defaults now.
2. Then we check how big the up queuesize should be to handle this rowset .
CQD EXE_MEMORY_FOR_UNPACK_ROWS_IN_MB is set to 100MB for this. In this case we dynamically adjust the queue size to accommodate larger rows. Eg is we have 100 rows of 2 MB each, we will limit upqueue size to 50 and allow 50 entries in upqueue.
I have changed the default value for this CQD to be 1GB to match (1.)
3. At runtime, the problem is that after passing these 2 checks, we try to allocate double the size of the up queue size for the full row length .
Assume we calculate in step(2) the upqueue size is N, then we allocate rowlength*N*2 at runtime. This clearly exceeds what we have estimated at compiletime.
This change includes a change in compiler to estimate the memory correctly.
So with these we can handle upto 16MB of long varchar/LOB data for batch input data a little better. But driver will still need to ensure that the rowset buffer will fit into the 1GB limit otherwise an error will be returned at compiletime. So for batch input for large rows , we need to limit number of rows in the rowset.
Also included a fix for TRAFODION-909
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sandhyasun/trafodion traf_lob_global_fix
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafodion/pull/1503.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1503
----
commit 528e9c60345234e554d32db637c82728b63c945b
Author: Sandhya Sundaresan <sa...@...>
Date: 2018-03-29T05:56:15Z
Performance changes for LOB and increasing limit for larger varchars/bobs to use with batch input.
----
---
[GitHub] trafodion pull request #1503: [TRAFOODION-2949] Performance changes for LOB ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/trafodion/pull/1503
---
[GitHub] trafodion pull request #1503: [TRAFOODION-2949] Performance changes for LOB ...
Posted by sandhyasun <gi...@git.apache.org>.
Github user sandhyasun commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1503#discussion_r179604511
--- Diff: core/sql/exp/exp_attrs.h ---
@@ -584,9 +587,10 @@ class Attributes : public NAVersionedObject
CASEINSENSITIVE_ = 0x0400, // caseinsensitive char/varchar datatype
- FORCE_FIXED_ = 0x0800 // Force this attribute to be treated as fixed
+ FORCE_FIXED_ = 0x0800, // Force this attribute to be treated as fixed
// in an aligned row. Used by HashGroupby for
// varchar aggregates
+ LENGTH_IN_KB_ =0x1000 // Indicates length is in KB
--- End diff --
It's in bytes.
---
[GitHub] trafodion pull request #1503: [TRAFOODION-2949] Performance changes for LOB ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/trafodion/pull/1503#discussion_r179534230
--- Diff: core/sql/exp/exp_attrs.h ---
@@ -584,9 +587,10 @@ class Attributes : public NAVersionedObject
CASEINSENSITIVE_ = 0x0400, // caseinsensitive char/varchar datatype
- FORCE_FIXED_ = 0x0800 // Force this attribute to be treated as fixed
+ FORCE_FIXED_ = 0x0800, // Force this attribute to be treated as fixed
// in an aligned row. Used by HashGroupby for
// varchar aggregates
+ LENGTH_IN_KB_ =0x1000 // Indicates length is in KB
--- End diff --
What is the length if it is not in KB?
---