You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by anoopsharma00 <gi...@git.apache.org> on 2015/11/28 00:26:51 UTC
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
GitHub user anoopsharma00 opened a pull request:
https://github.com/apache/incubator-trafodion/pull/194
[TRAFODION-1645] add support to retrieve region stats through sql queries
Details in JIRA.
Examples of sql commands in testcase regress/seabase/TEST002,EXPECTED002
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/anoopsharma00/incubator-trafodion br_regstats
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/194.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #194
----
commit 51698645d910ee0eee1b9301caae3f743bbbba07
Author: Cloud User <ce...@ansharma-1.novalocal>
Date: 2015-11-25T20:28:24Z
JIRA Trafodion-1645 SQL commands to retrieve region stats.
Details in JIRA.
commit 0b8693459d37367a674dcafe96ad3eca65b3c353
Author: Cloud User <ce...@ansharma-1.novalocal>
Date: 2015-11-25T20:29:22Z
Merge remote branch 'origin/master' into br_regstats
commit 9e0a88b8efafa9eb67f9a32bb4e88912ad0fe53f
Author: Cloud User <ce...@ansharma-1.novalocal>
Date: 2015-11-26T16:17:41Z
Merge remote branch 'origin/master' into br_regstats
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/194#discussion_r46314687
--- Diff: core/sql/comexe/ComTdbExeUtil.h ---
@@ -3452,6 +3453,122 @@ class ComTdbExeUtilHBaseBulkUnLoad : public ComTdbExeUtil
NABasicPtr snapshotSuffix_; // 30 - 37
char fillersExeUtilHbaseUnLoad_[16]; // 38 - 53
};
+
+static const ComTdbVirtTableColumnInfo comTdbRegionStatsVirtTableColumnInfo[] =
+ {
+ { "CATALOG_NAME", 0, COM_USER_COLUMN, REC_BYTE_F_ASCII, 256, FALSE, SQLCHARSETCODE_UTF8 , 0, 0, 0, 0, 0, 0, 0, COM_NO_DEFAULT, "",NULL,NULL, COM_UNKNOWN_DIRECTION_LIT, 0 },
--- End diff --
Just curious: Why fixed length CHAR instead of VARCHAR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/194#discussion_r46317167
--- Diff: core/sql/parser/ParKeyWords.cpp ---
@@ -1030,6 +1031,7 @@ ParKeyWord ParKeyWords::keyWords_[] = {
ParKeyWord("STATEMENT", TOK_STATEMENT, NONRESTOKEN_),
ParKeyWord("STATIC", TOK_STATIC, NONRESTOKEN_),
ParKeyWord("STATISTICS", TOK_STATISTICS, SECOND_|NONRESTOKEN_),
+ ParKeyWord("STATS", TOK_STATS, NONRESTOKEN_),
--- End diff --
Should we use the STATISTICS keyword, since it is already available?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/194#discussion_r46316498
--- Diff: core/sql/executor/ExExeUtilGet.cpp ---
@@ -5315,4 +5358,745 @@ ExExeUtilHiveMDaccessPrivateState::~ExExeUtilHiveMDaccessPrivateState()
{
};
+///////////////////////////////////////////////////////////////////
+ex_tcb * ExExeUtilRegionStatsTdb::build(ex_globals * glob)
+{
+ ExExeUtilRegionStatsTcb * exe_util_tcb;
+
+ if (displayFormat())
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsFormatTcb(*this, glob);
+ else
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsTcb(*this, glob);
+
+ exe_util_tcb->registerSubtasks();
+
+ return (exe_util_tcb);
+}
+
+////////////////////////////////////////////////////////////////
+// Constructor for class ExExeUtilRegionStatsTcb
+///////////////////////////////////////////////////////////////
+ExExeUtilRegionStatsTcb::ExExeUtilRegionStatsTcb(
+ const ComTdbExeUtilRegionStats & exe_util_tdb,
+ ex_globals * glob)
+ : ExExeUtilTcb( exe_util_tdb, NULL, glob)
+{
+ statsBuf_ = new(glob->getDefaultHeap()) char[sizeof(ComTdbRegionStatsVirtTableColumnStruct)];
+ statsBufLen_ = sizeof(ComTdbRegionStatsVirtTableColumnStruct);
+
+ stats_ = (ComTdbRegionStatsVirtTableColumnStruct*)statsBuf_;
+
+ inputNameBuf_ = NULL;
+ if (exe_util_tdb.inputExpr_)
+ {
+ inputNameBuf_ = new(glob->getDefaultHeap()) char[exe_util_tdb.inputRowlen_];
+ }
+
+ int jniDebugPort = 0;
+ int jniDebugTimeout = 0;
+ ehi_ = ExpHbaseInterface::newInstance(glob->getDefaultHeap(),
+ (char*)"", //exe_util_tdb.server(),
+ (char*)"", //exe_util_tdb.zkPort(),
+ jniDebugPort,
+ jniDebugTimeout);
+
+ regionInfoList_ = NULL;
+
+ tableName_ = new(glob->getDefaultHeap()) char[2000];
+
+ // get hbase rootdir location
+ hbaseRootdir_ = new(glob->getDefaultHeap()) char[1000];
+ strcpy(hbaseRootdir_, "/hbase");
+
+ step_ = INITIAL_;
+}
+
+ExExeUtilRegionStatsTcb::~ExExeUtilRegionStatsTcb()
+{
+ if (statsBuf_)
+ NADELETEBASIC(statsBuf_, getGlobals()->getDefaultHeap());
+
+ if (ehi_)
+ delete ehi_;
+
+ statsBuf_ = NULL;
+}
+
+//////////////////////////////////////////////////////
+// work() for ExExeUtilRegionStatsTcb
+//////////////////////////////////////////////////////
+Int64 ExExeUtilRegionStatsTcb::getEmbeddedNumValue
+(char* &sep, char endChar, NABoolean adjustLen)
+{
+ Int64 num = -1;
+ char * sepEnd = strchr(sep+1, endChar);
+ if (sepEnd)
+ {
+ char longBuf[30];
+
+ Lng32 len = sepEnd - sep - 1;
+ str_cpy_all(longBuf, (sep+1), len);
+ longBuf[len] = 0;
+
+ num = str_atoi(longBuf, len);
+
+ sep += len + 1;
+
+ if ((adjustLen) && (num == 0))
+ num = 1024;
+ }
+
+ return num;
+}
+
+short ExExeUtilRegionStatsTcb::collectStats(char * tableName)
+{
+ // populate catName_, schName_, objName_.
+ if (extractParts(tableName,
+ &catName_, &schName_, &objName_))
+ {
+ return -1;
+ }
+
+ // collect stats from ehi.
+ HbaseStr tblName;
+
+ NAString extNameForHbase =
+ NAString(catName_) + "." + NAString(schName_) + "." + NAString(objName_);
+ tblName.val = (char*)extNameForHbase.data();
+ tblName.len = extNameForHbase.length();
+
+ regionInfoList_ = ehi_->getRegionStats(tblName);
+ if (! regionInfoList_)
+ {
+ return -1;
+ }
+
+ currIndex_ = 0;
+
+ return 0;
+}
+
+short ExExeUtilRegionStatsTcb::populateStats
+(Int32 currIndex, NABoolean nullTerminate)
+{
+ str_pad(stats_->catalogName, sizeof(stats_->catalogName), ' ');
+ str_cpy_all(stats_->catalogName, catName_, strlen(catName_));
+ if (nullTerminate)
+ stats_->catalogName[strlen(catName_)] = 0;
+
+ str_pad(stats_->schemaName, sizeof(stats_->schemaName), ' ');
+ str_cpy_all(stats_->schemaName, schName_, strlen(schName_));
+ if (nullTerminate)
+ stats_->schemaName[strlen(schName_)] = 0;
+
+ str_pad(stats_->objectName, sizeof(stats_->objectName), ' ');
+ str_cpy_all(stats_->objectName, objName_, strlen(objName_));
+ if (nullTerminate)
+ stats_->objectName[strlen(objName_)] = 0;
+
+ str_pad(stats_->regionName, sizeof(stats_->regionName), ' ');
+ stats_->regionNum = currIndex_+1;
+
+ char regionInfoBuf[5000];
+ Int32 len = 0;
+ char * regionInfo =
+ regionInfoList_->getEntry
+ (currIndex, regionInfoBuf, 5000, len);
+ regionInfo[len] = 0;
+
+ stats_->numStores = 0;
+ stats_->numStoreFiles = 0;
+ stats_->storeFileUncompSize = 0;
+ stats_->storeFileSize = 0;
+ stats_->memStoreSize = 0;
+
+ char longBuf[30];
+ char * sep1 = strchr(regionInfo, '|');
+ if (sep1)
+ {
+ str_cpy_all(stats_->regionName, regionInfo,
+ (Lng32)(sep1 - regionInfo));
+
+ if (nullTerminate)
+ stats_->regionName[sep1 - regionInfo] = 0;
+ }
+
+ char * sepStart = sep1;
+ stats_->numStores = getEmbeddedNumValue(sepStart, '|', FALSE);
--- End diff --
This code seems to be assuming a particular format of text output (from HBase I assume). What happens if that format changes? Seems like we don't detect that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/194#discussion_r46315973
--- Diff: core/sql/executor/ExExeUtilGet.cpp ---
@@ -5315,4 +5358,745 @@ ExExeUtilHiveMDaccessPrivateState::~ExExeUtilHiveMDaccessPrivateState()
{
};
+///////////////////////////////////////////////////////////////////
+ex_tcb * ExExeUtilRegionStatsTdb::build(ex_globals * glob)
+{
+ ExExeUtilRegionStatsTcb * exe_util_tcb;
+
+ if (displayFormat())
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsFormatTcb(*this, glob);
+ else
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsTcb(*this, glob);
+
+ exe_util_tcb->registerSubtasks();
+
+ return (exe_util_tcb);
+}
+
+////////////////////////////////////////////////////////////////
+// Constructor for class ExExeUtilRegionStatsTcb
+///////////////////////////////////////////////////////////////
+ExExeUtilRegionStatsTcb::ExExeUtilRegionStatsTcb(
+ const ComTdbExeUtilRegionStats & exe_util_tdb,
+ ex_globals * glob)
+ : ExExeUtilTcb( exe_util_tdb, NULL, glob)
+{
+ statsBuf_ = new(glob->getDefaultHeap()) char[sizeof(ComTdbRegionStatsVirtTableColumnStruct)];
+ statsBufLen_ = sizeof(ComTdbRegionStatsVirtTableColumnStruct);
+
+ stats_ = (ComTdbRegionStatsVirtTableColumnStruct*)statsBuf_;
+
+ inputNameBuf_ = NULL;
+ if (exe_util_tdb.inputExpr_)
+ {
+ inputNameBuf_ = new(glob->getDefaultHeap()) char[exe_util_tdb.inputRowlen_];
+ }
+
+ int jniDebugPort = 0;
+ int jniDebugTimeout = 0;
+ ehi_ = ExpHbaseInterface::newInstance(glob->getDefaultHeap(),
+ (char*)"", //exe_util_tdb.server(),
+ (char*)"", //exe_util_tdb.zkPort(),
+ jniDebugPort,
+ jniDebugTimeout);
+
+ regionInfoList_ = NULL;
+
+ tableName_ = new(glob->getDefaultHeap()) char[2000];
+
+ // get hbase rootdir location
+ hbaseRootdir_ = new(glob->getDefaultHeap()) char[1000];
--- End diff --
Path lengths in Linux can be 1024; will 1000 be enough?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/194#discussion_r46316310
--- Diff: core/sql/executor/ExExeUtilGet.cpp ---
@@ -5315,4 +5358,745 @@ ExExeUtilHiveMDaccessPrivateState::~ExExeUtilHiveMDaccessPrivateState()
{
};
+///////////////////////////////////////////////////////////////////
+ex_tcb * ExExeUtilRegionStatsTdb::build(ex_globals * glob)
+{
+ ExExeUtilRegionStatsTcb * exe_util_tcb;
+
+ if (displayFormat())
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsFormatTcb(*this, glob);
+ else
+ exe_util_tcb = new(glob->getSpace()) ExExeUtilRegionStatsTcb(*this, glob);
+
+ exe_util_tcb->registerSubtasks();
+
+ return (exe_util_tcb);
+}
+
+////////////////////////////////////////////////////////////////
+// Constructor for class ExExeUtilRegionStatsTcb
+///////////////////////////////////////////////////////////////
+ExExeUtilRegionStatsTcb::ExExeUtilRegionStatsTcb(
+ const ComTdbExeUtilRegionStats & exe_util_tdb,
+ ex_globals * glob)
+ : ExExeUtilTcb( exe_util_tdb, NULL, glob)
+{
+ statsBuf_ = new(glob->getDefaultHeap()) char[sizeof(ComTdbRegionStatsVirtTableColumnStruct)];
+ statsBufLen_ = sizeof(ComTdbRegionStatsVirtTableColumnStruct);
+
+ stats_ = (ComTdbRegionStatsVirtTableColumnStruct*)statsBuf_;
+
+ inputNameBuf_ = NULL;
+ if (exe_util_tdb.inputExpr_)
+ {
+ inputNameBuf_ = new(glob->getDefaultHeap()) char[exe_util_tdb.inputRowlen_];
+ }
+
+ int jniDebugPort = 0;
+ int jniDebugTimeout = 0;
+ ehi_ = ExpHbaseInterface::newInstance(glob->getDefaultHeap(),
+ (char*)"", //exe_util_tdb.server(),
+ (char*)"", //exe_util_tdb.zkPort(),
+ jniDebugPort,
+ jniDebugTimeout);
+
+ regionInfoList_ = NULL;
+
+ tableName_ = new(glob->getDefaultHeap()) char[2000];
+
+ // get hbase rootdir location
+ hbaseRootdir_ = new(glob->getDefaultHeap()) char[1000];
+ strcpy(hbaseRootdir_, "/hbase");
+
+ step_ = INITIAL_;
+}
+
+ExExeUtilRegionStatsTcb::~ExExeUtilRegionStatsTcb()
+{
+ if (statsBuf_)
+ NADELETEBASIC(statsBuf_, getGlobals()->getDefaultHeap());
+
+ if (ehi_)
+ delete ehi_;
+
+ statsBuf_ = NULL;
+}
+
+//////////////////////////////////////////////////////
+// work() for ExExeUtilRegionStatsTcb
+//////////////////////////////////////////////////////
+Int64 ExExeUtilRegionStatsTcb::getEmbeddedNumValue
+(char* &sep, char endChar, NABoolean adjustLen)
+{
+ Int64 num = -1;
+ char * sepEnd = strchr(sep+1, endChar);
+ if (sepEnd)
+ {
+ char longBuf[30];
+
+ Lng32 len = sepEnd - sep - 1;
+ str_cpy_all(longBuf, (sep+1), len);
+ longBuf[len] = 0;
+
+ num = str_atoi(longBuf, len);
+
+ sep += len + 1;
+
+ if ((adjustLen) && (num == 0))
+ num = 1024;
+ }
+
+ return num;
+}
+
+short ExExeUtilRegionStatsTcb::collectStats(char * tableName)
+{
+ // populate catName_, schName_, objName_.
+ if (extractParts(tableName,
+ &catName_, &schName_, &objName_))
+ {
+ return -1;
+ }
+
+ // collect stats from ehi.
+ HbaseStr tblName;
+
+ NAString extNameForHbase =
+ NAString(catName_) + "." + NAString(schName_) + "." + NAString(objName_);
+ tblName.val = (char*)extNameForHbase.data();
+ tblName.len = extNameForHbase.length();
+
+ regionInfoList_ = ehi_->getRegionStats(tblName);
+ if (! regionInfoList_)
+ {
+ return -1;
+ }
+
+ currIndex_ = 0;
+
+ return 0;
+}
+
+short ExExeUtilRegionStatsTcb::populateStats
+(Int32 currIndex, NABoolean nullTerminate)
+{
+ str_pad(stats_->catalogName, sizeof(stats_->catalogName), ' ');
+ str_cpy_all(stats_->catalogName, catName_, strlen(catName_));
+ if (nullTerminate)
+ stats_->catalogName[strlen(catName_)] = 0;
--- End diff --
We declared this field as 256 bytes; should we have declared them 257 to allow for the null?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request: [TRAFODION-1645] add support to ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/194
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---