You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by selvaganesang <gi...@git.apache.org> on 2016/12/11 22:22:33 UTC
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
GitHub user selvaganesang opened a pull request:
https://github.com/apache/incubator-trafodion/pull/876
[Trafodion 2351] Bulk load with log error rows enhancements
The LOAD command now outputs as follows:
Load with log error rows into selva.customer select * from hive.hive.customer ;
Task: LOAD Status: Started Object: TRAFODION.SELVA.CUSTOMER
Task: CLEANUP Status: Started Time: 2016-12-11 00:54:37.642
Task: CLEANUP Status: Ended Time: 2016-12-11 00:54:37.672
Task: CLEANUP Status: Ended Elapsed Time: 00:00:00.030
Logging Location: /bulkload/logs/ERR_TRAFODION.SELVA.CUSTOMER_20161211_005437
Task: LOADING DATA Status: Started Time: 2016-12-11 00:54:37.672
Rows Processed: 99997
Task: LOADING DATA Status: Ended Time: 2016-12-11 00:54:58.296
Task: LOADING DATA Status: Ended Elapsed Time: 00:00:20.624
Task: COMPLETION Status: Started Time: 2016-12-11 00:54:58.296
Task: COMPLETION Status: Ended Time: 2016-12-11 00:54:59.521
Task: COMPLETION Status: Ended Elapsed Time: 00:00:00.756
In addition, currently the "LOADING DATA" task status is shown as it happens. In future,
the task status will be shown as it happens for all tasks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvaganesang/incubator-trafodion trafodion-2351
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-trafodion/pull/876.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #876
----
commit 055f7bb7ba344c7273e7ce2e725295a29f8768a2
Author: selvaganesang <se...@esgyn.com>
Date: 2016-11-18T22:12:57Z
Merge branch 'trafodion-2356' of github.com:selvaganesang/incubator-trafodion into trafodion-2356
commit e0dab3ee05c39b441de9f04fe3e04654b03cf8d4
Author: selvaganesang <se...@esgyn.com>
Date: 2016-11-19T00:40:11Z
Merge branch 'master' of git://git.apache.org/incubator-trafodion into trafodion-2356
commit 1c356227da4706b07962350eea6cfb9c9ad5304c
Author: selvaganesang <se...@esgyn.com>
Date: 2016-11-22T04:20:29Z
[TRAFODION-2356] HBase snapshot concept needs to be contained within
HBaseClient in Trafodion
Moved snapshot creation/deletion from SequenceFileWriter to HBaseClient.
commit 76cb8b4bf3f4f99acdeebb4f01c722af4079f1ff
Author: selvaganesang <se...@esgyn.com>
Date: 2016-12-08T21:32:53Z
Merge branch 'master' of git://git.apache.org/incubator-trafodion into trafodion-2351
Conflicts:
core/sql/executor/ExExeUtilLoad.cpp
core/sql/src/main/java/org/trafodion/sql/HBaseClient.java
commit e0244ddcafd6c7ce795cb6dd8f86df43695e7a2d
Author: selvaganesang <se...@esgyn.com>
Date: 2016-12-11T22:03:55Z
[TRAFODION-2351] Bulk load with log error rows enhancements
The LOAD command now outputs as follows:
Load with log error rows into selva.customer select * from hive.hive.customer ;
Task: LOAD Status: Started Object: TRAFODION.SELVA.CUSTOMER
Task: CLEANUP Status: Started Time: 2016-12-11 00:54:37.642
Task: CLEANUP Status: Ended Time: 2016-12-11 00:54:37.672
Task: CLEANUP Status: Ended Elapsed Time: 00:00:00.030
Logging Location: /bulkload/logs/ERR_TRAFODION.SELVA.CUSTOMER_20161211_005437
Task: LOADING DATA Status: Started Time: 2016-12-11 00:54:37.672
Rows Processed: 99997
Task: LOADING DATA Status: Ended Time: 2016-12-11 00:54:58.296
Task: LOADING DATA Status: Ended Elapsed Time: 00:00:20.624
Task: COMPLETION Status: Started Time: 2016-12-11 00:54:58.296
Task: COMPLETION Status: Ended Time: 2016-12-11 00:54:59.521
Task: COMPLETION Status: Ended Elapsed Time: 00:00:00.756
In addition, currently the "LOADING DATA" task status is shown as it happens. In future,
the task status will be shown as it happens for all tasks.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92966255
--- Diff: core/sql/executor/ExExeUtilCommon.cpp ---
@@ -641,8 +649,12 @@ short ExExeUtilTcb::executeQuery(char * task,
if (displayEndTime)
{
char timeBuf[200];
--- End diff --
Done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92281706
--- Diff: core/sql/executor/ExExeUtilCommon.cpp ---
@@ -641,8 +649,12 @@ short ExExeUtilTcb::executeQuery(char * task,
if (displayEndTime)
{
char timeBuf[200];
--- End diff --
Perhaps you want to delete this declaration, since it looks like you moved it up to line 558 in the new file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92283036
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -3033,14 +3033,23 @@ void ExHbaseAccessTcb::setRowID(char *rowId, Lng32 rowIdLen)
memcpy(rowID_.val, rowId, rowIdLen);
}
-void ExHbaseAccessTcb::buildLoggingPath(
- const char * loggingLocation,
- char * logId,
- const char * tableName,
+void ExHbaseAccessTcb::buildLoggingFileName(
+ const char * currCmdLoggingLocation,
+ const char *tableName,
const char * loggingFileNamePrefix,
Lng32 instId,
char * loggingFileName)
{
+ sprintf(loggingFileName, "%s/%s_%s_%d",
--- End diff --
How do we know loggingFileName is big enough to store the result?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92966327
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -3033,14 +3033,23 @@ void ExHbaseAccessTcb::setRowID(char *rowId, Lng32 rowIdLen)
memcpy(rowID_.val, rowId, rowIdLen);
}
-void ExHbaseAccessTcb::buildLoggingPath(
- const char * loggingLocation,
- char * logId,
- const char * tableName,
+void ExHbaseAccessTcb::buildLoggingFileName(
+ const char * currCmdLoggingLocation,
+ const char *tableName,
const char * loggingFileNamePrefix,
Lng32 instId,
char * loggingFileName)
{
+ sprintf(loggingFileName, "%s/%s_%s_%d",
--- End diff --
Yes. There is a possibility of buffer overflow because the size of the buffer is not calculated. Will fix it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [TRAFODION-2351] Bulk load with log e...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-trafodion/pull/876
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92966401
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -3050,9 +3059,9 @@ void ExHbaseAccessTcb::buildLoggingPath(
struct tm * curgmtime = gmtime(&t);
strftime(logId_tmp, sizeof(logId_tmp)-1, "%Y%m%d_%H%M%S", curgmtime);
logId = logId_tmp;
- }
- sprintf(loggingFileName, "%s/ERR_%s/%s_%s_%d",
- loggingLocation, logId, tableName, loggingFileNamePrefix, instId);
+ }
+ sprintf(currCmdLoggingLocation, "%s/ERR_%s_%s",
--- End diff --
The loggingLocation_ is allocated in line 1889 in setLoggingLocation_ with large enough buffer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92282189
--- Diff: core/sql/executor/ExExeUtilLoad.cpp ---
@@ -1778,6 +1811,19 @@ short ExExeUtilHBaseBulkLoadTcb::moveRowToUpQueue(const char * row, Lng32 len,
}
+short ExExeUtilHBaseBulkLoadTcb::printLoggingLocation(int bufPos)
+{
+ short retBufPos = bufPos;
+ if (hblTdb().getNoOutput())
+ return 0;
+ if (loggingLocation_ != NULL) {
+ str_sprintf(&statusMsgBuf_[bufPos], " Logging Location: %s", loggingLocation_);
--- End diff --
How do we know the target buffer is long enough? Is there some limit on the length of loggingLocation_?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92283123
--- Diff: core/sql/executor/ExHbaseAccess.cpp ---
@@ -3050,9 +3059,9 @@ void ExHbaseAccessTcb::buildLoggingPath(
struct tm * curgmtime = gmtime(&t);
strftime(logId_tmp, sizeof(logId_tmp)-1, "%Y%m%d_%H%M%S", curgmtime);
logId = logId_tmp;
- }
- sprintf(loggingFileName, "%s/ERR_%s/%s_%s_%d",
- loggingLocation, logId, tableName, loggingFileNamePrefix, instId);
+ }
+ sprintf(currCmdLoggingLocation, "%s/ERR_%s_%s",
--- End diff --
How do we know currCmdLoggingLocation is big enough to store the result?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-trafodion pull request #876: [Trafodion 2351] Bulk load with log e...
Posted by selvaganesang <gi...@git.apache.org>.
Github user selvaganesang commented on a diff in the pull request:
https://github.com/apache/incubator-trafodion/pull/876#discussion_r92965724
--- Diff: core/sql/executor/ExExeUtilLoad.cpp ---
@@ -1778,6 +1811,19 @@ short ExExeUtilHBaseBulkLoadTcb::moveRowToUpQueue(const char * row, Lng32 len,
}
+short ExExeUtilHBaseBulkLoadTcb::printLoggingLocation(int bufPos)
+{
+ short retBufPos = bufPos;
+ if (hblTdb().getNoOutput())
+ return 0;
+ if (loggingLocation_ != NULL) {
+ str_sprintf(&statusMsgBuf_[bufPos], " Logging Location: %s", loggingLocation_);
--- End diff --
The loggingLocation_ is allocated in line 1889 in setLoggingLocation_ with large enough buffer.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---