You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by selvaganesang <gi...@git.apache.org> on 2016/12/28 15:07:57 UTC
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
GitHub user selvaganesang opened a pull request:
https://github.com/apache/incubator-trafodion/pull/898
[TRAFODION-2420] RMS enhancements
Introduced a feature in RMS offender to list queries blocked in storage engine.
This lists queries which are blocked in the storage engine APIs
for more than given number of seconds. The BLOCKED_FOR_SECS denotes how
long the storage engine API is blocked.
The SQL commands to list SE_BLOCKED queries are available at:
$TRAF_HOME/export/limited-support-tools/LSO/se_blocked.sql
Example output is below:
cd $TRAF_HOME/export/limited-support-tools/LSO
./offender -s se_blocked
CURRENT_TIMESTAMP NO_OF_PROCESSES BLOCKED_FOR_SECS QUERY_ID TABLE_NAME
-------------------------- --------------- ---------------- ------------------------------------------------------------ ------------------------------
2016-12-28 10:29:43.941455 1 83 MXID11000030514212349680799580002000000000206U3333300_18_S1 TRAFODION.SCH.T022
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvaganesang/incubator-trafodion se_offender
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/898.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #898
----
commit ce6b44612f3fc19a18314f985853322dd39b66e4
Author: selvaganesang <se...@esgyn.com>
Date: 2016-12-28T14:57:49Z
[TRAFODION-2420] RMS enhancements
Introduced a feature to list queries blocked in storage engine
via RMS offender.
This lists queries which are blocked in the storage engine APIs
for more than given number of seconds. The BLOCKED_FOR_SECS denotes how
long the storage engine API is blocked.
The SQL commands to list SE_BLOCKED queries are available at:
$TRAF_HOME/export/limited-support-tools/LSO/se_blocked.sql
Example output is below:
cd $TRAF_HOME/export/limited-support-tools/LSO
./offender -s se_blocked
CURRENT_TIMESTAMP NO_OF_PROCESSES BLOCKED_FOR_SECS QUERY_ID TABLE_NAME
-------------------------- --------------- ---------------- ------------------------------------------------------------ ------------------------------
2016-12-28 10:29:43.941455 1 83 MXID11000030514212349680799580002000000000206U3333300_18_S1 TRAFODION.SCH.T022
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94860650
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
I think Arvind was correct. The number should be 25. For me this caused a regression failure by outputting a couple spurious stats at the end. (an extra Accessed Rows stat where none should be)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by svarnau <gi...@git.apache.org>.
Github user svarnau commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94868189
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Yes, that was the one!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94633228
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer,
"AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ",
ansiName_, fileName_,
bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(),
- fsDp2MsgsStats()->getNumMessages(),
- fsDp2MsgsStats()->getMessageBytes(),
- fsDp2MsgsStats()->getStatsBytes(),
+ exeSEStats()->getNumIOCalls(),
+ exeSEStats()->getNumIOBytes(),
--- End diff --
Thanks Arvind. I have fixed the issue with ExPartitionAccess::getVariableStatsInfo to provide 0 for statsBytes. ExPartitionAccess is not used currently. Hence, the regression tests didn't detect this issue. Thanks for catching it. I have left ExPartitionAccess behind because it might be resurrected later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94867234
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Where did you see this regression? Is it in core/TESTRTS?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94578723
--- Diff: core/conn/odbc/src/odbc/nsksrvrcore/ResStatisticsStatement.cpp ---
@@ -1265,7 +1265,7 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY
#define MAX_PERTABLE_STATS_DESC 30
#define MAX_MASTERSTATS_ENTRY 31
-#define MAX_MEASSTATS_ENTRY 34
+#define MAX_MEASSTATS_ENTRY 26
#define MAX_PERTABLE_ENTRY 14
--- End diff --
Yes. Changed it. Missed out in ODBC code but changed it in Type2 JDBC driver. I think there is a need to share this code between all components SQL, JDBC and ODBC.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94448449
--- Diff: core/conn/jdbc_type2/native/ResStatisticsStatement.cpp ---
@@ -1537,39 +1537,31 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY
initSqlStatsItems(measStatsItems_, MAX_MEASSTATS_ENTRY, &sqlStatsDesc_[currStatsDescEntry_]);
measStatsItems_[0].statsItem_id = SQLSTATS_ACT_ROWS_ACCESSED;
measStatsItems_[1].statsItem_id = SQLSTATS_ACT_ROWS_USED;
- measStatsItems_[2].statsItem_id = SQLSTATS_MSG_COUNT;
- measStatsItems_[3].statsItem_id = SQLSTATS_MSG_BYTES;
- measStatsItems_[4].statsItem_id = SQLSTATS_STATS_BYTES;
- measStatsItems_[5].statsItem_id = SQLSTATS_DISK_IOS;
- measStatsItems_[6].statsItem_id = SQLSTATS_LOCK_WAITS;
- measStatsItems_[7].statsItem_id = SQLSTATS_LOCK_ESCALATIONS;
- measStatsItems_[8].statsItem_id = SQLSTATS_DP2_CPU_BUSY_TIME;
- measStatsItems_[9].statsItem_id = SQLSTATS_SQL_CPU_BUSY_TIME;
- measStatsItems_[10].statsItem_id = SQLSTATS_SQL_SPACE_ALLOC;
- measStatsItems_[11].statsItem_id = SQLSTATS_SQL_SPACE_USED;
- measStatsItems_[12].statsItem_id = SQLSTATS_SQL_HEAP_ALLOC;
- measStatsItems_[13].statsItem_id = SQLSTATS_SQL_HEAP_USED;
- measStatsItems_[14].statsItem_id = SQLSTATS_EID_SPACE_ALLOC;
- measStatsItems_[15].statsItem_id = SQLSTATS_EID_SPACE_USED;
- measStatsItems_[16].statsItem_id = SQLSTATS_EID_HEAP_ALLOC;
- measStatsItems_[17].statsItem_id = SQLSTATS_EID_HEAP_USED;
- measStatsItems_[18].statsItem_id = SQLSTATS_OPENS;
- measStatsItems_[19].statsItem_id = SQLSTATS_OPEN_TIME;
- measStatsItems_[20].statsItem_id = SQLSTATS_PROCESS_CREATED;
- measStatsItems_[21].statsItem_id = SQLSTATS_PROCESS_CREATE_TIME;
- measStatsItems_[22].statsItem_id = SQLSTATS_REQ_MSG_CNT;
- measStatsItems_[23].statsItem_id = SQLSTATS_REQ_MSG_BYTES;
- measStatsItems_[24].statsItem_id = SQLSTATS_REPLY_MSG_CNT;
- measStatsItems_[25].statsItem_id = SQLSTATS_REPLY_MSG_BYTES;
- measStatsItems_[26].statsItem_id = SQLSTATS_SCRATCH_FILE_COUNT;
- measStatsItems_[27].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCK_SIZE;
- measStatsItems_[28].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ;
- measStatsItems_[29].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN;
- measStatsItems_[30].statsItem_id = SQLSTATS_SCRATCH_OVERFLOW_MODE;
- measStatsItems_[31].statsItem_id = SQLSTATS_SCRATCH_READ_COUNT;
- measStatsItems_[32].statsItem_id = SQLSTATS_SCRATCH_WRITE_COUNT;
- measStatsItems_[33].statsItem_id = SQLSTATS_UDR_CPU_BUSY_TIME;
- // MAX_MEASSTATS_ENTRY is set to 34
+ measStatsItems_[2].statsItem_id = SQLSTATS_SE_IOS;
--- End diff --
Looks like you took all the old DP2-specific counters out and added some "SE" counters. Is my understanding correct?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94878400
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
It looks like I looked at the wrong file ResStatisticsStatement.cpp when I responded to Arvind. Hence I didn't catch the issue with duplicate index 22
measStatsItems_[22].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_READ;
measStatsItems_[22].statsItem_id = SQLSTATS_SCRATCH_BUFFER_BLOCKS_WRITTEN; measStatsItems_[23].statsItem_id = SQLSTATS_SCRATCH_READ_COUNT;
measStatsItems_[24].statsItem_id = SQLSTATS_SCRATCH_WRITE_COUNT;
I will fix it
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by arvind-narain <gi...@git.apache.org>.
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94629541
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer,
"AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ",
ansiName_, fileName_,
bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(),
- fsDp2MsgsStats()->getNumMessages(),
- fsDp2MsgsStats()->getMessageBytes(),
- fsDp2MsgsStats()->getStatsBytes(),
+ exeSEStats()->getNumIOCalls(),
+ exeSEStats()->getNumIOBytes(),
--- End diff --
Thanks Selva. Noticed that we have 10 format specifiers (including StatsBytes) and only 9 arguments hence the question. Line 4522 has similar logic ( though it's sprintf). Maybe I'm missing something here - will check on the code again as well as str_sprintf.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94578756
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
No, it should be 26
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by arvind-narain <gi...@git.apache.org>.
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94495473
--- Diff: core/conn/odbc/src/odbc/nsksrvrcore/ResStatisticsStatement.cpp ---
@@ -1265,7 +1265,7 @@ void ResStatisticsStatement::setStatistics(SRVR_STMT_HDL *pSrvrStmt, SQLSTATS_TY
#define MAX_PERTABLE_STATS_DESC 30
#define MAX_MASTERSTATS_ENTRY 31
-#define MAX_MEASSTATS_ENTRY 34
+#define MAX_MEASSTATS_ENTRY 26
#define MAX_PERTABLE_ENTRY 14
--- End diff --
Should MAX_PERTABLE_ENTRY here be 10 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by arvind-narain <gi...@git.apache.org>.
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94497176
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer,
"AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ",
ansiName_, fileName_,
bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(),
- fsDp2MsgsStats()->getNumMessages(),
- fsDp2MsgsStats()->getMessageBytes(),
- fsDp2MsgsStats()->getStatsBytes(),
+ exeSEStats()->getNumIOCalls(),
+ exeSEStats()->getNumIOBytes(),
--- End diff --
0 for StatsBytes ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by arvind-narain <gi...@git.apache.org>.
Github user arvind-narain commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94496467
--- Diff: core/sql/executor/ExExeUtilGetStats.cpp ---
@@ -2319,47 +2321,38 @@ short ExExeUtilGetRTSStatisticsTcb::work()
{
if (measStatsItems_ == NULL)
{
- maxMeasStatsItems_ = 35;
+ maxMeasStatsItems_ = 26;
--- End diff --
Should this be 25 or 26 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/898#discussion_r94579003
--- Diff: core/sql/executor/ExStats.cpp ---
@@ -3678,9 +3292,8 @@ void ExPartitionAccessStats::getVariableStatsInfo(char * dataBuffer,
"AnsiName: %s PhysName: %s BuffersSize: %u BuffersSent: %u BuffersRcvd: %u NumMessages: %Ld MsgBytes: %Ld StatsBytes: %Ld MsgBytesSent: %Ld MsgBytesRcvd: %Ld ",
ansiName_, fileName_,
bufferStats()->sendBufferSize(), bufferStats()->sentBuffers().entryCnt(), bufferStats()->recdBuffers().entryCnt(),
- fsDp2MsgsStats()->getNumMessages(),
- fsDp2MsgsStats()->getMessageBytes(),
- fsDp2MsgsStats()->getStatsBytes(),
+ exeSEStats()->getNumIOCalls(),
+ exeSEStats()->getNumIOBytes(),
--- End diff --
Yes. It is supposed to denote how many stats bytes are transferred from SE to the executor periodically. But with the new SE, this concept is no longer applicable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #898: [TRAFODION-2420] RMS enhancements
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/898
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---