You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@trafodion.apache.org by su...@apache.org on 2017/03/15 15:52:50 UTC
[1/2] incubator-trafodion git commit: Various fixes, details below
Repository: incubator-trafodion
Updated Branches:
refs/heads/master 2ba737c55 -> eea1054ee
Various fixes, details below
-- max length limited to 16777216 bytes (16M)
for char cols and functions (repeat, concat).
(optimizer/SynthType.cpp, common/ComSmallDefs.h, sqlcomp/nadefaults.cpp)
-- previous max length change requires adding of a new hbase property
called hbase.client.keyvalue.maxsize so large key/values could be
handled in hbase cell.
Following scripts have been updated to handle that.
(sqf/sql/scripts/install_local_hadoop,
install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/
service_advisor.py
install/installer/traf_hortonworks_mods
install/python-installer/configs/mod_cfgs.json)
Developers can also update hbase-site.xml with this property if
they dont want to reinstall local hadoop.
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>0</value>
</property>
-- while accessing a hive table as an external table, the hive table and
corresponding external table definitions are validated to be the
same. This validates that corresponding columns have the same
data attributes (type, length, scale, etc).
This check causes failures if hive column is of 'string' datatype.
That is because hive 'string' column length can be changed by
a cqd but the corresponding external table has predefined length
set when the table is created.
The validation check now ignores the length attribute if the hive column is
of 'string' datatype.
(optimizer/BindRelExpr.cpp, common/CharType.*, NAType.*)
-- data moved into direct buffer would sometimes cause overflow and crash.
The max direct buffer length used to send/retrieve hbase data
is now limited to 1G (executor/ExHbaseAccess.cpp)
-- errors during vsbb upsert are now handled correctly
(executor/ExHbaseIUD.cpp)
-- support for GET CATALOGS command
(generator/GenRelExeUtil.cpp, executor/ExExeUtilGet.cpp)
-- An incorrect computation would sometimes cause group by rollup to
crash compiler in NAHeap::unlinkLargeFragment()
(generator/GenRelGrby.cpp)
-- sort of a large row that exceeded sort pre-set buffer size would crash
if the row size exceeded the max buffer size.
That has been fixed by allocating space for atleast one row.
(generator/GenRelMisc.cpp)
-- purgedata on a delimited name was failing. That has been fixed.
(optimizer/RelExeUtil.h)
-- regress/tools/runregr_privs1/privs2 fixed to handle running of
subset of tests
-- regress/seabase/TEST031 updated with new tests
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/8aa53233
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/8aa53233
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/8aa53233
Branch: refs/heads/master
Commit: 8aa532330e4dbf316a09817a033292c85cb02c4e
Parents: 03fd040
Author: Anoop Sharma <an...@esgyn.com>
Authored: Tue Mar 14 20:16:08 2017 +0000
Committer: Anoop Sharma <an...@esgyn.com>
Committed: Tue Mar 14 20:16:08 2017 +0000
----------------------------------------------------------------------
core/sqf/sql/scripts/install_local_hadoop | 4 +
core/sql/bin/SqlciErrors.txt | 2 +-
core/sql/common/CharType.cpp | 41 +++-
core/sql/common/CharType.h | 7 +-
core/sql/common/ComSmallDefs.h | 4 +
core/sql/common/NAType.cpp | 11 +
core/sql/common/NAType.h | 3 +
core/sql/executor/ExExeUtilGet.cpp | 14 ++
core/sql/executor/ExHbaseAccess.cpp | 18 +-
core/sql/executor/ExHbaseIUD.cpp | 30 ++-
core/sql/generator/GenRelExeUtil.cpp | 3 +-
core/sql/generator/GenRelGrby.cpp | 10 +-
core/sql/generator/GenRelMisc.cpp | 10 +-
core/sql/optimizer/BindRelExpr.cpp | 40 +++-
core/sql/optimizer/ItemExpr.h | 1 -
core/sql/optimizer/RelExeUtil.h | 2 +-
core/sql/optimizer/SynthType.cpp | 16 +-
core/sql/regress/charsets/EXPECTED004 | 6 +-
core/sql/regress/hive/EXPECTED009 | 23 +-
core/sql/regress/hive/TEST009 | 3 +-
core/sql/regress/seabase/EXPECTED011 | 18 +-
core/sql/regress/seabase/EXPECTED031 | 230 ++++++++++++++++++-
core/sql/regress/seabase/TEST011 | 2 +
core/sql/regress/seabase/TEST031 | 74 ++++++
core/sql/regress/tools/runregr_privs1.ksh | 49 ++--
core/sql/regress/tools/runregr_privs2.ksh | 49 ++--
core/sql/sqlcomp/DefaultConstants.h | 7 +-
core/sql/sqlcomp/nadefaults.cpp | 42 +++-
.../TRAFODION/2.1/service_advisor.py | 3 +-
install/installer/traf_hortonworks_mods | 6 +
install/python-installer/configs/mod_cfgs.json | 1 +
31 files changed, 595 insertions(+), 134 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sqf/sql/scripts/install_local_hadoop
----------------------------------------------------------------------
diff --git a/core/sqf/sql/scripts/install_local_hadoop b/core/sqf/sql/scripts/install_local_hadoop
index da5da45..a96bf31 100755
--- a/core/sqf/sql/scripts/install_local_hadoop
+++ b/core/sqf/sql/scripts/install_local_hadoop
@@ -1634,6 +1634,10 @@ echo "$MY_LOCAL_SW_DIST/${HBASE_TAR}"
</property>
${HREGION_IMPL}
${MASTER_VISBILITY_COPROC}
+ <property>
+ <name>hbase.client.keyvalue.maxsize</name>
+ <value>0</value>
+ </property>
</configuration>
EOF
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/bin/SqlciErrors.txt
----------------------------------------------------------------------
diff --git a/core/sql/bin/SqlciErrors.txt b/core/sql/bin/SqlciErrors.txt
index 69b628a..7e159cd 100644
--- a/core/sql/bin/SqlciErrors.txt
+++ b/core/sql/bin/SqlciErrors.txt
@@ -1198,7 +1198,7 @@ $1~String1 --------------------------------
4126 42000 99999 BEGINNER MAJOR DBADMIN The row value constructors in a VALUES clause must be of equal degree.
4127 42000 99999 BEGINNER MAJOR DBADMIN Type $0~String0 cannot be assigned to type $1~String1.
4128 42000 99999 BEGINNER MAJOR DBADMIN Default volume and subvolume information could not be retrieved from =_DEFAULTS define - DEFINEINFO error $0~int0.
-4129 ZZZZZ 99999 BEGINNER MAJOR DBADMIN The character-typed result of the function $0~String0 is longer than 32768 octets.
+4129 ZZZZZ 99999 BEGINNER MAJOR DBADMIN The character-typed result of the function $0~String0 is longer than max supported size.
4130 ZZZZZ 99999 BEGINNER MAJOR DBADMIN $0~string0 is a read-only DEFAULTS attribute and cannot be updated.
4131 42000 99999 BEGINNER MAJOR DBADMIN Current_time, current_date, or current_timestamp is not allowed in a check constraint.
4132 ZZZZZ 99999 BEGINNER MAJOR DBADMIN Current_user, session_user, or system_user is not allowed in a check constraint.
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/common/CharType.cpp
----------------------------------------------------------------------
diff --git a/core/sql/common/CharType.cpp b/core/sql/common/CharType.cpp
index 299fe56..aa0de61 100644
--- a/core/sql/common/CharType.cpp
+++ b/core/sql/common/CharType.cpp
@@ -752,16 +752,21 @@ NABoolean CharType::isCharSetAndCollationComboOK() const
// -----------------------------------------------------------------------
// Equality comparison
// -----------------------------------------------------------------------
+NABoolean CharType::compareCharAttrs(const NAType& other) const
+{
+ return caseinsensitive_ == ((CharType&)other).caseinsensitive_ &&
+ nullTerminated_ == ((CharType&)other).nullTerminated_ &&
+ upshifted_ == ((CharType&)other).upshifted_ &&
+ charSet_ == ((CharType&)other).charSet_ &&
+ collation_ == ((CharType&)other).collation_ &&
+ coercibility_ == ((CharType&)other).coercibility_ &&
+ encodingCharSet_ == ((CharType&)other).encodingCharSet_ ;
+}
+
NABoolean CharType::operator==(const NAType& other) const
{
return NAType::operator==(other) &&
- caseinsensitive_ == ((CharType&)other).caseinsensitive_ &&
- nullTerminated_ == ((CharType&)other).nullTerminated_ &&
- upshifted_ == ((CharType&)other).upshifted_ &&
- charSet_ == ((CharType&)other).charSet_ &&
- collation_ == ((CharType&)other).collation_ &&
- coercibility_ == ((CharType&)other).coercibility_ &&
- encodingCharSet_ == ((CharType&)other).encodingCharSet_ ;
+ compareCharAttrs(other);
}
// -----------------------------------------------------------------------
@@ -967,6 +972,28 @@ SQLVarChar::SQLVarChar(const CharLenInfo & maxLenInfo,
{}
// -----------------------------------------------------------------------
+// Equality comparison
+// -----------------------------------------------------------------------
+NABoolean SQLVarChar::operator==(const NAType& other) const
+{
+ NABoolean compResult = FALSE; // false, failed. true, passed.
+
+ // if this or other was originally a hive string type, then
+ // do not compare lengths.
+ if ((wasHiveString()) ||
+ ((other.getTypeName() == LiteralVARCHAR) &&
+ (((SQLVarChar&)other).wasHiveString())))
+ compResult = equalIgnoreLength(other);
+ else
+ compResult = NAType::operator==(other);
+
+ if (NOT compResult)
+ return FALSE;
+
+ return compareCharAttrs(other);
+}
+
+// -----------------------------------------------------------------------
// The SQL/C Preprocessor rewrites a VARCHAR declaration (Ansi 19.4)
// as an ANSIVARCHAR datatype token (Tandem-internal extension),
// which is actually this ANSIChar datatype --
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/common/CharType.h
----------------------------------------------------------------------
diff --git a/core/sql/common/CharType.h b/core/sql/common/CharType.h
index 3a026bd..44064de 100644
--- a/core/sql/common/CharType.h
+++ b/core/sql/common/CharType.h
@@ -250,7 +250,10 @@ virtual NAString* getKey(CollHeap* h=0) const; // compute hash key from SQL name
virtual NABoolean operator==(const NAType& other) const;
- virtual NABoolean isCompatible(const NAType& other, UInt32 * flags = NULL) const;
+// compares sttributes specific to char datatypes
+virtual NABoolean compareCharAttrs(const NAType& other) const;
+
+virtual NABoolean isCompatible(const NAType& other, UInt32 * flags = NULL) const;
#ifndef CLI_SRL
virtual NABoolean isComparable(const NAType &other,
@@ -543,6 +546,8 @@ virtual double encode (void*) const;
virtual NABoolean isEncodingNeeded() const { return TRUE; }
+virtual NABoolean operator==(const NAType& other) const;
+
// ---------------------------------------------------------------------
// A virtual function to return a copy of the type.
// ---------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/common/ComSmallDefs.h
----------------------------------------------------------------------
diff --git a/core/sql/common/ComSmallDefs.h b/core/sql/common/ComSmallDefs.h
index 417ab86..f20799c 100644
--- a/core/sql/common/ComSmallDefs.h
+++ b/core/sql/common/ComSmallDefs.h
@@ -1937,6 +1937,10 @@ enum ComQiScope
#define COM_TEXT__TEXT__MD_COL_MAX_SIZE 3000
#endif // ! defined(COM_TEXT__TEXT__MD_COL_MAX_SIZE)
+// max size 16Mb
+#define MAX_CHAR_COL_LENGTH_IN_BYTES 16777216
+#define MAX_CHAR_COL_LENGTH_IN_BYTES_STR "16777216"
+
//
// Definition of class ComUID
//
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/common/NAType.cpp
----------------------------------------------------------------------
diff --git a/core/sql/common/NAType.cpp b/core/sql/common/NAType.cpp
index 8ed7129..1753af8 100644
--- a/core/sql/common/NAType.cpp
+++ b/core/sql/common/NAType.cpp
@@ -225,6 +225,17 @@ NABoolean NAType::operator==(const NAType& other) const
return FALSE;
} // operator==()
+NABoolean NAType::equalIgnoreLength(const NAType& other) const
+{
+ if (typeName_ == other.typeName_ &&
+ qualifier_ == other.qualifier_ &&
+ SQLnullFlag_ == other.SQLnullFlag_ &&
+ varLenFlag_ == other.varLenFlag_)
+ return TRUE;
+ else
+ return FALSE;
+}
+
NABoolean NAType::equalIgnoreNull(const NAType& other) const
{
if (typeName_ == other.typeName_ &&
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/common/NAType.h
----------------------------------------------------------------------
diff --git a/core/sql/common/NAType.h b/core/sql/common/NAType.h
index 06755ea..238aa4d 100644
--- a/core/sql/common/NAType.h
+++ b/core/sql/common/NAType.h
@@ -166,6 +166,9 @@ public:
// Identical as NAType::operator==() except null attribute is not compared.
virtual NABoolean equalIgnoreNull(const NAType& other) const;
+ // Identical as NAType::operator==() except length attribute is not compared.
+ virtual NABoolean equalIgnoreLength(const NAType& other) const;
+
// Checks NATypes to see if they are equal just like the operator==
// except for the SQLnullFlag_ flags. For the SQLnullFlag_ it
// checks to see if it's physical.
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/executor/ExExeUtilGet.cpp
----------------------------------------------------------------------
diff --git a/core/sql/executor/ExExeUtilGet.cpp b/core/sql/executor/ExExeUtilGet.cpp
index bba6ce7..5e2d70a 100644
--- a/core/sql/executor/ExExeUtilGet.cpp
+++ b/core/sql/executor/ExExeUtilGet.cpp
@@ -266,6 +266,12 @@ static const QueryString getComponentPrivileges[] =
{" ; "}
};
+static const QueryString getCatalogsQuery[] =
+{
+ {" select * from (values ('TRAFODION'), ('HIVE')) "},
+ {" order by 1 desc "},
+ {" ; "}
+};
static const QueryString getTrafTablesInSchemaQuery[] =
{
@@ -1628,6 +1634,14 @@ short ExExeUtilGetMetadataInfoTcb::work()
switch (getMItdb().queryType_)
{
+ case ComTdbExeUtilGetMetadataInfo::CATALOGS_:
+ {
+ qs = getCatalogsQuery;
+ sizeOfqs = sizeof(getCatalogsQuery);
+
+ }
+ break;
+
case ComTdbExeUtilGetMetadataInfo::TABLES_IN_SCHEMA_:
{
qs = getTrafTablesInSchemaQuery;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/executor/ExHbaseAccess.cpp
----------------------------------------------------------------------
diff --git a/core/sql/executor/ExHbaseAccess.cpp b/core/sql/executor/ExHbaseAccess.cpp
index e931959..02f8223 100644
--- a/core/sql/executor/ExHbaseAccess.cpp
+++ b/core/sql/executor/ExHbaseAccess.cpp
@@ -2634,8 +2634,22 @@ void ExHbaseAccessTcb::allocateDirectRowBufferForJNI(
4 )) // for col value len
+ numCols; // 1 byte null value
maxRowLen = rowLen + directBufferOverhead;
- directRowBufferLen_ = (maxRowLen * maxRows);
- directBufferMaxRows_ = maxRows;
+
+ // limit direct buffer len to 1G
+ Int64 tempDirectBuflen = (Int64)maxRowLen * (Int64)maxRows;
+ Int64 maxDirectBuflen = 1024*1024*1024;
+
+ if (tempDirectBuflen > maxDirectBuflen)
+ {
+ directRowBufferLen_ = MINOF(tempDirectBuflen, maxDirectBuflen);
+ directBufferMaxRows_ = (Int64)directRowBufferLen_ / (Int64)maxRowLen;
+ }
+ else
+ {
+ directRowBufferLen_ = (maxRowLen * maxRows);
+ directBufferMaxRows_ = maxRows;
+ }
+
directRowBuffer_ = new (getHeap()) BYTE[directRowBufferLen_];
rows_.val = (char *)directRowBuffer_;
rows_.len = 0;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/executor/ExHbaseIUD.cpp
----------------------------------------------------------------------
diff --git a/core/sql/executor/ExHbaseIUD.cpp b/core/sql/executor/ExHbaseIUD.cpp
index 0b8414d..ddb63ca 100644
--- a/core/sql/executor/ExHbaseIUD.cpp
+++ b/core/sql/executor/ExHbaseIUD.cpp
@@ -763,19 +763,27 @@ ExWorkProcRetcode ExHbaseAccessUpsertVsbbSQTcb::work()
nextRequest_ = qparent_.down->getHeadIndex();
ex_queue_entry *pentry_down = qparent_.down->getHeadEntry();
- if (pentry_down->downState.request == ex_queue::GET_NOMORE)
+ if ((step_ == HANDLE_ERROR) ||
+ (step_ == CLOSE_AND_DONE))
+ {
+ // move down to error/close case.
+ }
+ else if (pentry_down->downState.request == ex_queue::GET_NOMORE)
step_ = CLOSE_AND_DONE;
else if (pentry_down->downState.request == ex_queue::GET_EOD)
- if (currRowNum_ > rowsInserted_)
- {
- step_ = PROCESS_INSERT_AND_CLOSE;
- }
- else
- {
- if (lastHandledStep_ == ALL_DONE)
- matches_=0;
- step_ = ALL_DONE;
- }
+ {
+ if (currRowNum_ > rowsInserted_)
+ {
+ step_ = PROCESS_INSERT_AND_CLOSE;
+ }
+ else
+ {
+ if (lastHandledStep_ == ALL_DONE)
+ matches_=0;
+ step_ = ALL_DONE;
+ }
+ }
+
switch (step_)
{
case NOT_STARTED:
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/generator/GenRelExeUtil.cpp
----------------------------------------------------------------------
diff --git a/core/sql/generator/GenRelExeUtil.cpp b/core/sql/generator/GenRelExeUtil.cpp
index 5318dc9..9e70bc5 100644
--- a/core/sql/generator/GenRelExeUtil.cpp
+++ b/core/sql/generator/GenRelExeUtil.cpp
@@ -1980,8 +1980,7 @@ short ExeUtilGetMetadataInfo::codeGen(Generator * generator)
(infoType_ == "MVGROUPS") ||
(infoType_ == "SYNONYMS") ||
(infoType_ == "PARTITIONS") ||
- (infoType_ == "TRIGGERS") ||
- (infoType_ == "CATALOGS"))
+ (infoType_ == "TRIGGERS"))
{
NAString nas("GET ");
nas += infoType_;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/generator/GenRelGrby.cpp
----------------------------------------------------------------------
diff --git a/core/sql/generator/GenRelGrby.cpp b/core/sql/generator/GenRelGrby.cpp
index e1e3c65..7a778e4 100644
--- a/core/sql/generator/GenRelGrby.cpp
+++ b/core/sql/generator/GenRelGrby.cpp
@@ -105,9 +105,13 @@ short GroupByAgg::genAggrGrbyExpr(Generator * generator,
ValueId valId;
// find the number of aggregate and groupby entries
- ULng32 numAttrs
- = ((NOT aggregateExpr.isEmpty() ) ? aggregateExpr.entries() : 0)
- + ((NOT groupExpr.isEmpty() ) ? groupExpr.entries() : 0);
+ ULng32 numAttrs = 0;
+ if (NOT aggregateExpr.isEmpty())
+ numAttrs += aggregateExpr.entries();
+ if (isRollup() && (NOT rollupGroupExprList.isEmpty()))
+ numAttrs += rollupGroupExprList.entries();
+ else if (NOT groupExpr.isEmpty())
+ numAttrs += groupExpr.entries();
NABoolean isAggrOneRow_ = FALSE;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/generator/GenRelMisc.cpp
----------------------------------------------------------------------
diff --git a/core/sql/generator/GenRelMisc.cpp b/core/sql/generator/GenRelMisc.cpp
index a89e0cb..99eaa85 100644
--- a/core/sql/generator/GenRelMisc.cpp
+++ b/core/sql/generator/GenRelMisc.cpp
@@ -3045,8 +3045,11 @@ short Sort::generateTdb(Generator * generator,
CostScalar bufferSize = getDefault(GEN_SORT_MAX_BUFFER_SIZE);
- ULng32 bufferSize_as_ulong =
- (ULng32)(MINOF(CostScalar(UINT_MAX), bufferSize)).getValue();
+ UInt32 bufferSize_as_ulong =
+ (UInt32)(MINOF(CostScalar(UINT_MAX), bufferSize)).getValue();
+
+ // allocate buffer to hold atlease one row
+ bufferSize_as_ulong = MAXOF(bufferSize_as_ulong, sortRecLen);
GenAssert(sortRecLen <= bufferSize_as_ulong,
"Record Len greater than GEN_SORT_MAX_BUFFER_SIZE");
@@ -3079,6 +3082,9 @@ short Sort::generateTdb(Generator * generator,
if(sortRecLen >= sort_options->scratchIOBlockSize())
{
Int32 maxScratchIOBlockSize = (Int32)getDefault(SCRATCH_IO_BLOCKSIZE_SORT_MAX);
+ // allocate space for atleast one row.
+ maxScratchIOBlockSize = MAXOF(maxScratchIOBlockSize, sortRecLen);
+
GenAssert(sortRecLen <= maxScratchIOBlockSize,
"sortRecLen is greater than SCRATCH_IO_BLOCKSIZE_SORT_MAX");
sort_options->scratchIOBlockSize() = MINOF(sortRecLen * 128, maxScratchIOBlockSize);
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/optimizer/BindRelExpr.cpp
----------------------------------------------------------------------
diff --git a/core/sql/optimizer/BindRelExpr.cpp b/core/sql/optimizer/BindRelExpr.cpp
index bf206c2..8d81cd6 100644
--- a/core/sql/optimizer/BindRelExpr.cpp
+++ b/core/sql/optimizer/BindRelExpr.cpp
@@ -1666,18 +1666,36 @@ NATable *BindWA::getNATable(CorrName& corrName,
return NULL;
}
- // Compare column lists
+ // Compare native and external table definitions.
+ // -- If this call is to drop external table, skip comparison.
+ // -- Otherwise compare that number of columns is the same.
+ // -- Compare type for corresponding columns. But if external table
+ // was created with explicit col attrs, then skip type check for cols.
+ //
// TBD - return what mismatches
- if ( nativeNATable &&
- !(table->getNAColumnArray() == nativeNATable->getNAColumnArray()) &&
- (NOT bindWA->externalTableDrop()))
- {
- *CmpCommon::diags() << DgSqlCode(-3078)
- << DgString0(adjustedName)
- << DgTableName(table->getTableName().getQualifiedNameAsAnsiString());
- bindWA->setErrStatus();
- nativeNATable->setRemoveFromCacheBNC(TRUE);
- return NULL;
+ NABoolean compError = FALSE;
+ if (nativeNATable &&
+ (NOT bindWA->externalTableDrop()))
+ {
+ if (table->getNAColumnArray().entries() !=
+ nativeNATable->getNAColumnArray().entries())
+ compError = TRUE;
+ if ((NOT compError) &&
+ (NOT table->hiveExtColAttrs()) &&
+ (NOT table->hiveExtKeyAttrs()))
+ {
+ if (NOT (table->getNAColumnArray() == nativeNATable->getNAColumnArray()))
+ compError = TRUE;
+ }
+ if (compError)
+ {
+ *CmpCommon::diags() << DgSqlCode(-3078)
+ << DgString0(adjustedName)
+ << DgTableName(table->getTableName().getQualifiedNameAsAnsiString());
+ bindWA->setErrStatus();
+ nativeNATable->setRemoveFromCacheBNC(TRUE);
+ return NULL;
+ }
}
}
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/optimizer/ItemExpr.h
----------------------------------------------------------------------
diff --git a/core/sql/optimizer/ItemExpr.h b/core/sql/optimizer/ItemExpr.h
index 9ad7035..6e5e57a 100644
--- a/core/sql/optimizer/ItemExpr.h
+++ b/core/sql/optimizer/ItemExpr.h
@@ -73,7 +73,6 @@ class VEGRewritePairs;
class ColStatDescList;
#define CONST_32K 32768
-#define CONST_100K 100000
// -----------------------------------------------------------------------
// An ExprValueId object is a pointer to an ItemExpr object. The pointer
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/optimizer/RelExeUtil.h
----------------------------------------------------------------------
diff --git a/core/sql/optimizer/RelExeUtil.h b/core/sql/optimizer/RelExeUtil.h
index 03630b7..1e7ef4c 100644
--- a/core/sql/optimizer/RelExeUtil.h
+++ b/core/sql/optimizer/RelExeUtil.h
@@ -402,7 +402,7 @@ public:
short ddlXnsInfo(NABoolean &ddlXns, NABoolean &xnCanBeStarted);
- NAString getQualObjName() { return qualObjName_.getQualifiedNameAsString(); }
+ NAString getQualObjName() { return qualObjName_.getQualifiedNameAsAnsiString(); }
void setCreateMDViews(NABoolean v)
{(v ? flags_ |= CREATE_MD_VIEWS : flags_ &= ~CREATE_MD_VIEWS); }
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/optimizer/SynthType.cpp
----------------------------------------------------------------------
diff --git a/core/sql/optimizer/SynthType.cpp b/core/sql/optimizer/SynthType.cpp
index 474a537..537f73c 100644
--- a/core/sql/optimizer/SynthType.cpp
+++ b/core/sql/optimizer/SynthType.cpp
@@ -3860,14 +3860,15 @@ const NAType *Repeat::synthesizeType()
Int64 size_in_bytes;
Int64 size_in_chars;
+ Int32 maxCharColLen = CmpCommon::getDefaultNumeric(TRAF_MAX_CHARACTER_COL_LENGTH);
+
// figure out the max length of result.
NABoolean negate;
if (maxLengthWasExplicitlySet_)
{
// cap max len at traf_max_character_col_length
size_in_bytes =
- MINOF(CmpCommon::getDefaultNumeric(TRAF_MAX_CHARACTER_COL_LENGTH),
- getMaxLength());
+ MINOF(maxCharColLen, getMaxLength());
size_in_chars =
size_in_bytes / CharInfo::minBytesPerChar(ctyp1.getCharSet());
}
@@ -3898,21 +3899,21 @@ const NAType *Repeat::synthesizeType()
size_in_chars = ctyp1.getStrCharLimit() * repeatCount;
// check size limit only for fixed character type
if ( ! typ1.isVaryingLen() ) {
- if ( size_in_bytes > CONST_100K ) {
+ if ( size_in_bytes > maxCharColLen ) {
*CmpCommon::diags() << DgSqlCode(-4129)
<< DgString0(getTextUpper());
return NULL;
}
} else // varchar. The nominal size of the result is
- // the min of (size, CONST_100K).
+ // the min of (size, maxCharColLen).
{
- size_in_bytes = MINOF(CONST_100K, size_in_bytes);
+ size_in_bytes = MINOF(maxCharColLen, size_in_bytes);
size_in_chars = size_in_bytes / CharInfo::minBytesPerChar(ctyp1.getCharSet());
}
}
else if (getMaxLength() > -1)
{
- size_in_bytes = MINOF(CmpCommon::getDefaultNumeric(TRAF_MAX_CHARACTER_COL_LENGTH),
+ size_in_bytes = MINOF(maxCharColLen,
getMaxLength() * typ1.getNominalSize());
size_in_chars = size_in_bytes / CharInfo::minBytesPerChar(ctyp1.getCharSet());
}
@@ -3920,8 +3921,7 @@ const NAType *Repeat::synthesizeType()
{
// Assign some arbitrary max result size since we can't
// figure out the actual max size.
- size_in_bytes =
- CmpCommon::getDefaultNumeric(TRAF_MAX_CHARACTER_COL_LENGTH);
+ size_in_bytes = maxCharColLen;
size_in_chars = size_in_bytes / CharInfo::minBytesPerChar(ctyp1.getCharSet());
}
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/charsets/EXPECTED004
----------------------------------------------------------------------
diff --git a/core/sql/regress/charsets/EXPECTED004 b/core/sql/regress/charsets/EXPECTED004
index 079534d..b740bf8 100755
--- a/core/sql/regress/charsets/EXPECTED004
+++ b/core/sql/regress/charsets/EXPECTED004
@@ -435,7 +435,7 @@ select _ucs2 x'0a$#' from (values(1))x;
>>--large length values should not cause core-dump (case 10-040218-2346)
>>select space(2147483647,ucs2) from (values (1)) as x;
-*** ERROR[4129] The character-typed result of the function REPEAT is longer than 32768 octets.
+*** ERROR[4129] The character-typed result of the function REPEAT is longer than max supported size.
*** ERROR[4062] The preceding error actually occurred in function SPACE.
@@ -443,13 +443,13 @@ select _ucs2 x'0a$#' from (values(1))x;
>>select rpad(N'test',2147483647,N'T') from (values(1)) as x;
-*** ERROR[4129] The character-typed result of the function RPAD is longer than 32768 octets.
+*** ERROR[4129] The character-typed result of the function RPAD is longer than max supported size.
*** ERROR[8822] The statement was not prepared.
>>select lpad(N'test',2147483647,N'T') from (values(1)) as x;
-*** ERROR[4129] The character-typed result of the function REPEAT is longer than 32768 octets.
+*** ERROR[4129] The character-typed result of the function REPEAT is longer than max supported size.
*** ERROR[4062] The preceding error actually occurred in function LPAD.
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/hive/EXPECTED009
----------------------------------------------------------------------
diff --git a/core/sql/regress/hive/EXPECTED009 b/core/sql/regress/hive/EXPECTED009
index 900ff3b..d039a32 100644
--- a/core/sql/regress/hive/EXPECTED009
+++ b/core/sql/regress/hive/EXPECTED009
@@ -1,7 +1,8 @@
>>obey TEST009(set_up);
>>prepare get_hv_schemas from
+>select schema_name from "_MD_".objects
-+>where schema_name like '\_HV\_%' escape '\' and object_type = 'PS';
++>where (schema_name = '_HV_HIVE_' or schema_name = '_HV_SCH_T009_')
++> and object_type = 'PS';
--- SQL command prepared.
>>
@@ -346,7 +347,7 @@ T009T2
>>invoke hive.sch_t009.t009t1;
-- Definition of hive table T009T1
--- Definition current Tue Jan 31 21:32:51 2017
+-- Definition current Fri Feb 17 17:26:07 2017
(
A INT
@@ -416,7 +417,7 @@ _HB__CELL__
>>invoke trafodion."_HB__ROW__"."baseball";
-- Definition of Trafodion external table TRAFODION."_HB__ROW__"."baseball"
--- Definition current Tue Jan 31 21:33:04 2017
+-- Definition current Fri Feb 17 17:26:17 2017
(
ROW_ID VARCHAR(100) CHARACTER SET ISO88591
@@ -429,7 +430,7 @@ _HB__CELL__
>>invoke trafodion."_HB__CELL__"."baseball";
-- Definition of Trafodion external table TRAFODION."_HB__CELL__"."baseball"
--- Definition current Tue Jan 31 21:33:06 2017
+-- Definition current Fri Feb 17 17:26:18 2017
(
ROW_ID VARCHAR(100) CHARACTER SET ISO88591
@@ -518,7 +519,7 @@ ROW_ID COLS
>>invoke hive.hive.store_sales;
-- Definition of hive table STORE_SALES
--- Definition current Tue Jan 31 21:33:17 2017
+-- Definition current Fri Feb 17 17:26:28 2017
(
SS_SOLD_DATE_SK INT
@@ -562,7 +563,7 @@ ROW_ID COLS
------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... S
-PLAN_ID .................. 212352658397193063
+PLAN_ID .................. 212354112388392066
ROWS_OUT ............. 1,709
EST_TOTAL_COST ........... 0.27
STATEMENT ................ select * from store_sales where ss_item_sk = 1;
@@ -691,7 +692,7 @@ LC RC OP OPERATOR OPT DESCRIPTION CARD
------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... S
-PLAN_ID .................. 212352658398567771
+PLAN_ID .................. 212354112389908739
ROWS_OUT ......... 2,920,643
EST_TOTAL_COST ........... 1.07
STATEMENT ................ select *
@@ -893,7 +894,7 @@ DESCRIPTION
>>invoke hive.hive.date_dim;
-- Definition of hive table DATE_DIM
--- Definition current Tue Jan 31 21:33:22 2017
+-- Definition current Fri Feb 17 17:26:34 2017
(
D_DATE_SK INT
@@ -1034,7 +1035,7 @@ CREATE EXTERNAL TABLE DATE_DIM
------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... S
-PLAN_ID .................. 212352658405327436
+PLAN_ID .................. 212354112397694640
ROWS_OUT ................. 1
EST_TOTAL_COST ........... 0.01
STATEMENT ................ select *
@@ -1125,7 +1126,7 @@ DESCRIPTION
>>invoke hive.hive.date_dim;
-- Definition of hive table DATE_DIM
--- Definition current Tue Jan 31 21:33:32 2017
+-- Definition current Fri Feb 17 17:26:45 2017
(
D_DATE_SK INT
@@ -1266,7 +1267,7 @@ CREATE EXTERNAL TABLE DATE_DIM
------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... S
-PLAN_ID .................. 212352658415800073
+PLAN_ID .................. 212354112408875042
ROWS_OUT ................. 1
EST_TOTAL_COST ........... 0.01
STATEMENT ................ select *
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/hive/TEST009
----------------------------------------------------------------------
diff --git a/core/sql/regress/hive/TEST009 b/core/sql/regress/hive/TEST009
index 520cf75..e71467b 100755
--- a/core/sql/regress/hive/TEST009
+++ b/core/sql/regress/hive/TEST009
@@ -97,7 +97,8 @@ execute get_hb_schemas;
?section set_up
prepare get_hv_schemas from
select schema_name from "_MD_".objects
-where schema_name like '\_HV\_%' escape '\' and object_type = 'PS';
+where (schema_name = '_HV_HIVE_' or schema_name = '_HV_SCH_T009_')
+ and object_type = 'PS';
prepare get_hb_schemas from
select schema_name from "_MD_".objects
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/seabase/EXPECTED011
----------------------------------------------------------------------
diff --git a/core/sql/regress/seabase/EXPECTED011 b/core/sql/regress/seabase/EXPECTED011
index ff29b17..bd28f64 100644
--- a/core/sql/regress/seabase/EXPECTED011
+++ b/core/sql/regress/seabase/EXPECTED011
@@ -7,7 +7,7 @@
>>invoke T011T1;
-- Definition of Trafodion table TRAFODION.SCH.T011T1
--- Definition current Tue Oct 18 19:38:05 2016
+-- Definition current Fri Feb 17 17:11:37 2017
(
A INT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -143,7 +143,7 @@ A B
------------------------------------------------------------------ PLAN SUMMARY
MODULE_NAME .............. DYNAMICALLY COMPILED
STATEMENT_NAME ........... NOT NAMED
-PLAN_ID .................. 212343579486936982
+PLAN_ID .................. 212354111499137048
ROWS_OUT ................. 7
EST_TOTAL_COST ........... 0.01
STATEMENT ................ select * from t011t1 where a >= 3 and a < 5;
@@ -182,7 +182,7 @@ DESCRIPTION
QUERY_CACHE ............ 0
TRAF_ALIGNED_ROW_FORMAT OFF
GENERATE_EXPLAIN ....... ON
- ObjectUIDs ............. 971200116663334701
+ ObjectUIDs ............. 1451969908907031756
select_list ............ TRAFODION.SCH.T011T1.A, TRAFODION.SCH.T011T1.B
@@ -1204,6 +1204,9 @@ A B
>>
>>
>>-- tests for large columns
+>>cqd traf_max_character_col_length '200000';
+
+--- SQL operation complete.
>>create table t011t4 (a char(200000), b varchar(200000), c char(200000 bytes) character set utf8, d varchar(200000 bytes) character set utf8);
--- SQL operation complete.
@@ -1252,6 +1255,9 @@ a bbbbbbbbbb c dddd
*** ERROR[4247] Specified size in bytes (1000001) exceeds the maximum size allowed (200000) for column D.
--- SQL operation failed with errors.
+>>cqd traf_max_character_col_length reset;
+
+--- SQL operation complete.
>>
>>-- tests for repository explain
>>-- check for repository tables
@@ -1269,7 +1275,7 @@ METRIC_TEXT_TABLE
>>invoke trafodion."_REPOS_".metric_query_table;
-- Definition of Trafodion table TRAFODION."_REPOS_".METRIC_QUERY_TABLE
--- Definition current Tue Oct 18 19:40:14 2016
+-- Definition current Fri Feb 17 17:13:41 2017
(
INSTANCE_ID INT UNSIGNED NO DEFAULT NOT NULL NOT
@@ -1457,7 +1463,7 @@ METRIC_TEXT_TABLE
>>-- get qid for the prepared stmt
>>get qid for statement explstmt;
-MXID11000018109212343579451270139000000000206U3333300_2285_EXPLSTMT
+MXID11000018963212354111473302000000000000206U3333300_2378_EXPLSTMT
--- SQL operation complete.
>>
@@ -1509,7 +1515,7 @@ SEQ_NUM OPERATOR
--- SQL command prepared.
>>get qid for statement explstmt2;
-MXID11000018109212343579451270139000000000206U3333300_2296_EXPLSTMT2
+MXID11000018963212354111473302000000000000206U3333300_2389_EXPLSTMT2
--- SQL operation complete.
>>set qid MXID123456 for explstmt2;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/seabase/EXPECTED031
----------------------------------------------------------------------
diff --git a/core/sql/regress/seabase/EXPECTED031 b/core/sql/regress/seabase/EXPECTED031
index e5c580a..d3284b0 100644
--- a/core/sql/regress/seabase/EXPECTED031
+++ b/core/sql/regress/seabase/EXPECTED031
@@ -423,6 +423,78 @@ Z Z (EXPR) (EXPR)
--- SQL operation complete.
>>
+>>-- really large columns
+>>drop table if exists t031t10;
+
+--- SQL operation complete.
+>>
+>>-- should fail, exceeds max
+>>create table t031t10 (a varchar(20485760));
+
+*** ERROR[4247] Specified size in bytes (20485760) exceeds the maximum size allowed (1000000) for column A.
+
+--- SQL operation failed with errors.
+>>cqd traf_max_character_col_length_override 'ON';
+
+--- SQL operation complete.
+>>cqd traf_max_character_col_length '25000000';
+
+--- SQL operation complete.
+>>create table t031t10 (a varchar(20485760));
+
+--- SQL operation complete.
+>>invoke t031t10;
+
+-- Definition of Trafodion table TRAFODION.SCH.T031T10
+-- Definition current Tue Mar 14 19:41:33 2017
+
+ (
+ SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
+ , A VARCHAR(20485760) CHARACTER SET ISO88591
+ COLLATE DEFAULT DEFAULT NULL
+ )
+
+--- SQL operation complete.
+>>cqd traf_max_character_col_length_override 'OFF';
+
+--- SQL operation complete.
+>>
+>>-- should fail, exceeds max
+>>drop table if exists t031t11;
+
+--- SQL operation complete.
+>>create table t031t11 (a varchar(20485760));
+
+*** ERROR[4247] Specified size in bytes (20485760) exceeds the maximum size allowed (16777216) for column A.
+
+--- SQL operation failed with errors.
+>>create table t031t11 (a varchar(10485760));
+
+--- SQL operation complete.
+>>prepare s from select repeat('1234567890', 1048576 ) from dual;
+
+--- SQL command prepared.
+>>
+>>-- should fail, exceeds max
+>>prepare s from select repeat('11234567890', 1648576 ) from dual;
+
+*** ERROR[4129] The character-typed result of the function REPEAT is longer than max supported size.
+
+*** ERROR[8822] The statement was not prepared.
+
+>>
+>>-- should fail, exceeds max. TBD.
+>>prepare s from select ' ' || repeat('1234567890', 1048576) from dual;
+
+--- SQL command prepared.
+>>
+>>-- hive tables
+>>sh echo "drop table t031hive;" > TEST031_junk;
+>>sh regrhive.ksh -f TEST031_junk;
+>>
+>>sh echo "create table t031hive(z int, a string, b string);" > TEST031_junk;
+>>sh regrhive.ksh -f TEST031_junk;
+>>
>>cqd hive_max_string_length '1000000';
--- SQL operation complete.
@@ -576,7 +648,7 @@ LC RC OP OPERATOR OPT DESCRIPTION CARD
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:00 2017
+-- Definition current Tue Mar 14 19:42:19 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -607,7 +679,7 @@ A B
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:06 2017
+-- Definition current Tue Mar 14 19:42:25 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -658,7 +730,7 @@ A B C
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:14 2017
+-- Definition current Tue Mar 14 19:42:33 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -687,7 +759,7 @@ A B C
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:23 2017
+-- Definition current Tue Mar 14 19:42:41 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -712,7 +784,7 @@ A B C
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:28 2017
+-- Definition current Tue Mar 14 19:42:46 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -732,7 +804,7 @@ A B C
>>invoke t031t1;
-- Definition of Trafodion table TRAFODION.SCH.T031T1
--- Definition current Thu Feb 23 00:12:35 2017
+-- Definition current Tue Mar 14 19:42:53 2017
(
SYSKEY LARGEINT NO DEFAULT NOT NULL NOT DROPPABLE
@@ -747,7 +819,7 @@ A B C
>>invoke t031v1;
-- Definition of Trafodion view TRAFODION.SCH.T031V1
--- Definition current Thu Feb 23 00:12:38 2017
+-- Definition current Tue Mar 14 19:42:56 2017
(
A INT DEFAULT NULL
@@ -770,7 +842,7 @@ A B C
>>invoke t031v1;
-- Definition of Trafodion view TRAFODION.SCH.T031V1
--- Definition current Thu Feb 23 00:12:42 2017
+-- Definition current Tue Mar 14 19:43:00 2017
(
A INT DEFAULT NULL
@@ -788,7 +860,7 @@ A B C
>>invoke t031v1;
-- Definition of Trafodion view TRAFODION.SCH.T031V1
--- Definition current Thu Feb 23 00:12:52 2017
+-- Definition current Tue Mar 14 19:43:09 2017
(
A INT DEFAULT NULL
@@ -875,7 +947,7 @@ CREATE INDEX T031T1I1 ON TRAFODION.SCH.T031T1
>>invoke table(index_table t031t1i1);
-- Definition of Trafodion table TRAFODION.SCH.T031T1I1
--- Definition current Thu Feb 23 00:13:14 2017
+-- Definition current Tue Mar 14 19:43:33 2017
(
"A@" INT NO DEFAULT
@@ -888,6 +960,143 @@ CREATE INDEX T031T1I1 ON TRAFODION.SCH.T031T1
--- SQL operation complete.
>>
+>>-- purgedata of table with delimited name
+>>drop table if exists "tT";
+
+--- SQL operation complete.
+>>create table "tT" (a int);
+
+--- SQL operation complete.
+>>purgedata "tT";
+
+--- SQL operation complete.
+>>
+>>-- group by rollup would sometimes crash compiler.
+>>drop table if exists mytable;
+
+--- SQL operation complete.
+>>create table mytable(a char(10), b char(10), c int, d int);
+
+--- SQL operation complete.
+>>insert into mytable values
++>('A1', 'B1', 1, 1),
++>('A1', 'B1', 1, 2),
++>('A1', 'B1', 2, 3),
++>('A1', 'B1', 2, 4),
++>('A1', 'B2', 3, 5),
++>('A1', 'B2', 3, 6),
++>('A1', 'B2', 4, 7),
++>('A1', 'B2', 4, 8),
++>('A2', 'B3', 5, 9),
++>('A2', 'B3', 5, 10),
++>('A2', 'B3', 6, 11),
++>('A2', 'B3', 6, 12),
++>('A2', 'B4', 7, 13),
++>('A2', 'B4', 7, 14),
++>('A2', 'B4', 8, 15),
++>('A2', 'B4', 8, 16);
+
+--- 16 row(s) inserted.
+>>select a, b, c, sum(d) as newcol from mytable where a in ('A1')
++> group by(a, b, c);
+
+A B C NEWCOL
+---------- ---------- ----------- --------------------
+
+A1 B1 1 3
+A1 B2 3 11
+A1 B2 4 15
+A1 B1 2 7
+
+--- 4 row(s) selected.
+>>select a, b, c, sum(d) as newcol from mytable where d > 5
++> group by rollup(a, b, c);
+
+A B C NEWCOL
+---------- ---------- ----------- --------------------
+
+A1 B2 3 6
+A1 B2 4 15
+A1 B2 ? 21
+A1 ? ? 21
+A2 B3 5 19
+A2 B3 6 23
+A2 B3 ? 42
+A2 B4 7 27
+A2 B4 8 31
+A2 B4 ? 58
+A2 ? ? 100
+? ? ? 121
+
+--- 12 row(s) selected.
+>>select a, b, c, sum(d) as newcol from mytable where a in ('A1')
++> group by rollup(a, b, c);
+
+A B C NEWCOL
+---------- ---------- ----------- --------------------
+
+A1 B1 1 3
+A1 B1 2 7
+A1 B1 ? 10
+A1 B2 3 11
+A1 B2 4 15
+A1 B2 ? 26
+A1 B3 5 19
+A1 B3 6 23
+A1 B3 ? 42
+A1 B4 7 27
+A1 B4 8 31
+A1 B4 ? 58
+A1 ? ? 136
+A1 ? ? 136
+
+--- 14 row(s) selected.
+>>
+>>-- GET CATALOGS support
+>>get catalogs;
+
+Catalogs
+========
+
+TRAFODION
+HIVE
+
+--- SQL operation complete.
+>>
+>>-- external and hive table mismatch on hive 'string' datatype
+>>process hive statement 'drop table t031hivet1';
+
+--- SQL operation complete.
+>>process hive statement 'create table t031hivet1 (a string)';
+
+--- SQL operation complete.
+>>drop external table if exists t031hivet1 for hive.hive.t031hivet1;
+
+--- SQL operation complete.
+>>create external table t031hivet1 for hive.hive.t031hivet1;
+
+--- SQL operation complete.
+>>cqd hive_max_string_length '10';
+
+--- SQL operation complete.
+>>showddl hive.hive.t031hivet1;
+
+/* Hive DDL */
+CREATE TABLE T031HIVET1
+ (
+ A string
+ )
+ stored as textfile
+;
+
+/* Trafodion DDL */
+
+CREATE EXTERNAL TABLE T031HIVET1
+ FOR HIVE.HIVE.T031HIVET1
+;
+
+--- SQL operation complete.
+>>
>>-- volatile and external table operation in default hive schema
>>drop external table t031hive for hive.hive.t031hive;
@@ -915,4 +1124,5 @@ CREATE INDEX T031T1I1 ON TRAFODION.SCH.T031T1
--- SQL operation complete.
>>
+>>
>>log;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/seabase/TEST011
----------------------------------------------------------------------
diff --git a/core/sql/regress/seabase/TEST011 b/core/sql/regress/seabase/TEST011
index c690744..1c02774 100644
--- a/core/sql/regress/seabase/TEST011
+++ b/core/sql/regress/seabase/TEST011
@@ -297,6 +297,7 @@ select * from t011t1;
-- tests for large columns
+cqd traf_max_character_col_length '200000';
create table t011t4 (a char(200000), b varchar(200000), c char(200000 bytes) character set utf8, d varchar(200000 bytes) character set utf8);
insert into t011t4 values ('a', repeat('b', 200), 'c', repeat('d', 400));
select left (a, 20), left(b, 10), left(c, 10), left (d, 11) from t011t4;
@@ -310,6 +311,7 @@ select left (a, 20), left(b, 10), left(c, 10), left (d, 11) from t011t4;
-- negative test
create table t011t5 (a char(200000), b varchar(200000), c char(200000 bytes) character set utf8, d varchar(1000001 bytes) character set utf8);
+cqd traf_max_character_col_length reset;
-- tests for repository explain
-- check for repository tables
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/seabase/TEST031
----------------------------------------------------------------------
diff --git a/core/sql/regress/seabase/TEST031 b/core/sql/regress/seabase/TEST031
index e79b022..36593b2 100644
--- a/core/sql/regress/seabase/TEST031
+++ b/core/sql/regress/seabase/TEST031
@@ -153,6 +153,36 @@ select x.z, y.z, substring(x.a, 1, 5), substring(y.a,1,5)
process hive statement 'drop table t031hive';
process hive statement 'create table t031hive(z int, a string, b string)';
+-- really large columns
+drop table if exists t031t10;
+
+-- should fail, exceeds max
+create table t031t10 (a varchar(20485760));
+cqd traf_max_character_col_length_override 'ON';
+cqd traf_max_character_col_length '25000000';
+create table t031t10 (a varchar(20485760));
+invoke t031t10;
+cqd traf_max_character_col_length_override 'OFF';
+
+-- should fail, exceeds max
+drop table if exists t031t11;
+create table t031t11 (a varchar(20485760));
+create table t031t11 (a varchar(10485760));
+prepare s from select repeat('1234567890', 1048576 ) from dual;
+
+-- should fail, exceeds max
+prepare s from select repeat('11234567890', 1648576 ) from dual;
+
+-- should fail, exceeds max. TBD.
+prepare s from select ' ' || repeat('1234567890', 1048576) from dual;
+
+-- hive tables
+sh echo "drop table t031hive;" > TEST031_junk;
+sh regrhive.ksh -f TEST031_junk;
+
+sh echo "create table t031hive(z int, a string, b string);" > TEST031_junk;
+sh regrhive.ksh -f TEST031_junk;
+
cqd hive_max_string_length '1000000';
insert into hive.hive.t031hive select * from t031t1;
select count(*) from hive.hive.t031hive;
@@ -248,6 +278,49 @@ set parserflags 1;
invoke table(index_table t031t1i1);
reset parserflags;
+-- purgedata of table with delimited name
+drop table if exists "tT";
+create table "tT" (a int);
+purgedata "tT";
+
+-- group by rollup would sometimes crash compiler.
+drop table if exists mytable;
+create table mytable(a char(10), b char(10), c int, d int);
+insert into mytable values
+('A1', 'B1', 1, 1),
+('A1', 'B1', 1, 2),
+('A1', 'B1', 2, 3),
+('A1', 'B1', 2, 4),
+('A1', 'B2', 3, 5),
+('A1', 'B2', 3, 6),
+('A1', 'B2', 4, 7),
+('A1', 'B2', 4, 8),
+('A2', 'B3', 5, 9),
+('A2', 'B3', 5, 10),
+('A2', 'B3', 6, 11),
+('A2', 'B3', 6, 12),
+('A2', 'B4', 7, 13),
+('A2', 'B4', 7, 14),
+('A2', 'B4', 8, 15),
+('A2', 'B4', 8, 16);
+select a, b, c, sum(d) as newcol from mytable where a in ('A1')
+ group by(a, b, c);
+select a, b, c, sum(d) as newcol from mytable where d > 5
+ group by rollup(a, b, c);
+select a, b, c, sum(d) as newcol from mytable where a in ('A1')
+ group by rollup(a, b, c);
+
+-- GET CATALOGS support
+get catalogs;
+
+-- external and hive table mismatch on hive 'string' datatype
+process hive statement 'drop table t031hivet1';
+process hive statement 'create table t031hivet1 (a string)';
+drop external table if exists t031hivet1 for hive.hive.t031hivet1;
+create external table t031hivet1 for hive.hive.t031hivet1;
+cqd hive_max_string_length '10';
+showddl hive.hive.t031hivet1;
+
-- volatile and external table operation in default hive schema
drop external table t031hive for hive.hive.t031hive;
set schema hive.hive;
@@ -259,4 +332,5 @@ drop volatile table vtt;
create external table t031hive for t031hive;
drop external table t031hive for t031hive;
+
log;
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/tools/runregr_privs1.ksh
----------------------------------------------------------------------
diff --git a/core/sql/regress/tools/runregr_privs1.ksh b/core/sql/regress/tools/runregr_privs1.ksh
index 0f3e4a0..0d301ca 100755
--- a/core/sql/regress/tools/runregr_privs1.ksh
+++ b/core/sql/regress/tools/runregr_privs1.ksh
@@ -58,34 +58,37 @@ function usage {
#-----------------------------------------------------------------
diffsonly=0
dircleanup=0
-tests=
echo ""
-echo "RUNNING SEABASE PRIVS1 TESTS"
+echo "RUNNING PRIVS1 TESTS"
echo ""
#--------------------------------------------------------------
# Determine which command line arguments the user specified --
# and set the corresponding boolean variable to TRUE. --
#--------------------------------------------------------------
-while [ $# -gt 0 ]; do
- case $1 in
- -dircleanup)
- dircleanup=1
- ;;
-
- -diff)
- diffsonly=1
- ;;
-
- -h|-help)
- usage
- exit 0
- ;;
- *)
- tests="$1 "
- ;;
- esac
- shift
+OK=-1
+while [ $OK -ne 0 ]; do # loop to allow options to appear in any order
+
+ if [ $OK -gt 0 ]; then
+ shift $OK
+ fi
+ OK=0
+
+ if [ "$1" = "-dircleanup" ]; then
+ dircleanup=1
+ OK=1
+ fi
+
+ if [ "$1" = "-diff" ]; then
+ diffsonly=1
+ OK=1
+ fi
+
+ if [ "$1" = "-help" -o "$1" = "-h" ]; then
+ usage
+ exit 0
+ OK=1
+ fi
done
#########################################################################
@@ -163,11 +166,11 @@ done
# the tests beginnning with "TEST". --
# If only the number is specified, prepend TEST. --
#--------------------------------------------------
-if [ "$tests" = "" ]; then
+if [ "$1" = "" ]; then
testfiles=`ls -1 TEST???* | tr a-z A-Z | sed -e /~$/d -e /.bak$/d | sort -fu`
prettyfiles=$testfiles
else
- testfiles=`echo $tests | tr a-z A-Z`
+ testfiles=`echo $* | tr a-z A-Z`
prettyfiles=
for i in $testfiles; do
if [ `expr substr $i 1 4` = "TEST" ]; then
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/regress/tools/runregr_privs2.ksh
----------------------------------------------------------------------
diff --git a/core/sql/regress/tools/runregr_privs2.ksh b/core/sql/regress/tools/runregr_privs2.ksh
index 320295b..3e1e76b 100755
--- a/core/sql/regress/tools/runregr_privs2.ksh
+++ b/core/sql/regress/tools/runregr_privs2.ksh
@@ -58,34 +58,37 @@ function usage {
#-----------------------------------------------------------------
diffsonly=0
dircleanup=0
-tests=
echo ""
-echo "RUNNING SEABASE PRIVS2 TESTS"
+echo "RUNNING PRIVS2 TESTS"
echo ""
#--------------------------------------------------------------
# Determine which command line arguments the user specified --
# and set the corresponding boolean variable to TRUE. --
#--------------------------------------------------------------
-while [ $# -gt 0 ]; do
- case $1 in
- -dircleanup)
- dircleanup=1
- ;;
-
- -diff)
- diffsonly=1
- ;;
-
- -h|-help)
- usage
- exit 0
- ;;
- *)
- tests="$1 "
- ;;
- esac
- shift
+OK=-1
+while [ $OK -ne 0 ]; do # loop to allow options to appear in any order
+
+ if [ $OK -gt 0 ]; then
+ shift $OK
+ fi
+ OK=0
+
+ if [ "$1" = "-dircleanup" ]; then
+ dircleanup=1
+ OK=1
+ fi
+
+ if [ "$1" = "-diff" ]; then
+ diffsonly=1
+ OK=1
+ fi
+
+ if [ "$1" = "-help" -o "$1" = "-h" ]; then
+ usage
+ exit 0
+ OK=1
+ fi
done
#########################################################################
@@ -163,11 +166,11 @@ done
# the tests beginnning with "TEST". --
# If only the number is specified, prepend TEST. --
#--------------------------------------------------
-if [ "$tests" = "" ]; then
+if [ "$1" = "" ]; then
testfiles=`ls -1 TEST???* | tr a-z A-Z | sed -e /~$/d -e /.bak$/d | sort -fu`
prettyfiles=$testfiles
else
- testfiles=`echo $tests | tr a-z A-Z`
+ testfiles=`echo $* | tr a-z A-Z`
prettyfiles=
for i in $testfiles; do
if [ `expr substr $i 1 4` = "TEST" ]; then
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/sqlcomp/DefaultConstants.h
----------------------------------------------------------------------
diff --git a/core/sql/sqlcomp/DefaultConstants.h b/core/sql/sqlcomp/DefaultConstants.h
index a930d07..96c8f1e 100644
--- a/core/sql/sqlcomp/DefaultConstants.h
+++ b/core/sql/sqlcomp/DefaultConstants.h
@@ -3628,9 +3628,14 @@ enum DefaultConstants
TRAF_TRANS_TYPE,
- // max size in bytes of a char or varchar column in a trafodion table.
+ // max size in bytes of a char or varchar column in a trafodion table.
+ // Valid values are 0 through 10M (10485760).
TRAF_MAX_CHARACTER_COL_LENGTH,
+ // In special cases, previous default value could be overridden.
+ // Internal use only or use only under trafodion supervision.
+ TRAF_MAX_CHARACTER_COL_LENGTH_OVERRIDE,
+
// set when metadata definition is to be read from hardcoded structs
// and not from metadata.
TRAF_BOOTSTRAP_MD_MODE,
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/core/sql/sqlcomp/nadefaults.cpp
----------------------------------------------------------------------
diff --git a/core/sql/sqlcomp/nadefaults.cpp b/core/sql/sqlcomp/nadefaults.cpp
index ba9efdf..0814a15 100644
--- a/core/sql/sqlcomp/nadefaults.cpp
+++ b/core/sql/sqlcomp/nadefaults.cpp
@@ -3364,10 +3364,11 @@ XDDkwd__(SUBQUERY_UNNESTING, "ON"),
DDkwd__(TRAF_LOAD_USE_FOR_INDEXES, "ON"),
DDkwd__(TRAF_LOAD_USE_FOR_STATS, "OFF"),
- // max size in bytes of a char or varchar column.
- DDui2__(TRAF_MAX_CHARACTER_COL_LENGTH, "200000"),
+ // max size in bytes of a char or varchar column. Set to 16M
+ DDui___(TRAF_MAX_CHARACTER_COL_LENGTH, MAX_CHAR_COL_LENGTH_IN_BYTES_STR),
+ DDkwd__(TRAF_MAX_CHARACTER_COL_LENGTH_OVERRIDE, "OFF"),
- DDkwd__(TRAF_MULTI_COL_FAM, "ON"),
+ DDkwd__(TRAF_MULTI_COL_FAM, "ON"),
DDkwd__(TRAF_NO_CONSTR_VALIDATION, "OFF"),
@@ -5936,8 +5937,39 @@ enum DefaultConstants NADefaults::validateAndInsert(const char *attrName,
}
break;
- default: break;
- }
+ // max char col length is 10M (10485760).
+ // max char col length is defined in common/ComSmallDefs.h
+ // Currently set to 16M
+ // In special cases, it could be overridden. Internal use only or
+ // use only under trafodion supervision.
+ case TRAF_MAX_CHARACTER_COL_LENGTH:
+ {
+ NABoolean override = (getToken(TRAF_MAX_CHARACTER_COL_LENGTH_OVERRIDE) == DF_ON);
+ double d = atof(value.data());
+ if ((NOT override) &&
+ (NOT (d >= 0 && d <= MAX_CHAR_COL_LENGTH_IN_BYTES)))
+ {
+ *CmpCommon::diags() << DgSqlCode(-2055)
+ << DgString0(value)
+ << DgString1(lookupAttrName(attrEnum));
+ }
+ }
+ break;
+
+ case TRAF_MAX_CHARACTER_COL_LENGTH_OVERRIDE:
+ {
+ // if override is being turned off, reset max_char_len to default value.
+ if (value == "OFF")
+ {
+ NAString val;
+ validateAndInsert("TRAF_MAX_CHARACTER_COL_LENGTH", val, TRUE);
+ }
+ }
+ break;
+
+ default:
+ break;
+ }
} // code to valid overwrite (insert)
if (reset && overwrite) {
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/service_advisor.py
----------------------------------------------------------------------
diff --git a/install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/service_advisor.py b/install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/service_advisor.py
index 7488ce9..a007d0d 100644
--- a/install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/service_advisor.py
+++ b/install/ambari-installer/traf-mpack/common-services/TRAFODION/2.1/service_advisor.py
@@ -238,7 +238,8 @@ class TRAFODION21ServiceAdvisor(service_advisor.DefaultStackAdvisor):
"hbase.bulkload.staging.dir": "/apps/hbase/staging",
"hbase.regionserver.region.transactional.tlog": "true",
"hbase.snapshot.region.timeout": "600000",
- "hbase.client.scanner.timeout.period": "600000"
+ "hbase.client.scanner.timeout.period": "600000",
+ "hbase.client.keyvalue.maxsize": "0"
}
return desired
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/install/installer/traf_hortonworks_mods
----------------------------------------------------------------------
diff --git a/install/installer/traf_hortonworks_mods b/install/installer/traf_hortonworks_mods
index b451dce..2cca29e 100755
--- a/install/installer/traf_hortonworks_mods
+++ b/install/installer/traf_hortonworks_mods
@@ -262,6 +262,12 @@ if [ $? != 0 ]; then
exit -1
fi
sleep 2
+ssh -q -n $AMBARI_HOST "$AMBARI_DIR"'/configs.sh -u' "$ADMIN" '-p' "$PASSWORD" '-port' "$PORT" 'set' "$AMBARI_HOST" "$CLUSTER_NAME" 'hbase-site hbase.client.keyvalue.maxsize 0'
+if [ $? != 0 ]; then
+ echo "***ERROR: unable to modify hbase.client.keyvalue.maxsize through Ambari's configs.sh script."
+ exit -1
+fi
+sleep 2
echo
#Change Zookeeeper settings
http://git-wip-us.apache.org/repos/asf/incubator-trafodion/blob/8aa53233/install/python-installer/configs/mod_cfgs.json
----------------------------------------------------------------------
diff --git a/install/python-installer/configs/mod_cfgs.json b/install/python-installer/configs/mod_cfgs.json
index 5427c9c..2786c87 100644
--- a/install/python-installer/configs/mod_cfgs.json
+++ b/install/python-installer/configs/mod_cfgs.json
@@ -10,6 +10,7 @@
"hbase.regionserver.region.transactional.tlog": "true",
"hbase.snapshot.region.timeout": "600000",
"hbase.client.scanner.timeout.period": "600000"
+ "hbase.client.keyvalue.maxsize": "0"
},
"hdfs-site": { "dfs.namenode.acls.enabled": "true" },
"zoo.cfg": { "maxClientCnxns": "0" }
[2/2] incubator-trafodion git commit: Merge [TRAFODION-1702] PR-1009
Various fixes, details below
Posted by su...@apache.org.
Merge [TRAFODION-1702] PR-1009 Various fixes, details below
Project: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/commit/eea1054e
Tree: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/tree/eea1054e
Diff: http://git-wip-us.apache.org/repos/asf/incubator-trafodion/diff/eea1054e
Branch: refs/heads/master
Commit: eea1054ee6e400643e65cb14cc7445aa3ae76b07
Parents: 2ba737c 8aa5323
Author: Suresh Subbiah <su...@apache.org>
Authored: Wed Mar 15 15:52:19 2017 +0000
Committer: Suresh Subbiah <su...@apache.org>
Committed: Wed Mar 15 15:52:19 2017 +0000
----------------------------------------------------------------------
core/sqf/sql/scripts/install_local_hadoop | 4 +
core/sql/bin/SqlciErrors.txt | 2 +-
core/sql/common/CharType.cpp | 41 +++-
core/sql/common/CharType.h | 7 +-
core/sql/common/ComSmallDefs.h | 4 +
core/sql/common/NAType.cpp | 11 +
core/sql/common/NAType.h | 3 +
core/sql/executor/ExExeUtilGet.cpp | 14 ++
core/sql/executor/ExHbaseAccess.cpp | 18 +-
core/sql/executor/ExHbaseIUD.cpp | 30 ++-
core/sql/generator/GenRelExeUtil.cpp | 3 +-
core/sql/generator/GenRelGrby.cpp | 10 +-
core/sql/generator/GenRelMisc.cpp | 10 +-
core/sql/optimizer/BindRelExpr.cpp | 40 +++-
core/sql/optimizer/ItemExpr.h | 1 -
core/sql/optimizer/RelExeUtil.h | 2 +-
core/sql/optimizer/SynthType.cpp | 16 +-
core/sql/regress/charsets/EXPECTED004 | 6 +-
core/sql/regress/hive/EXPECTED009 | 23 +-
core/sql/regress/hive/TEST009 | 3 +-
core/sql/regress/seabase/EXPECTED011 | 18 +-
core/sql/regress/seabase/EXPECTED031 | 230 ++++++++++++++++++-
core/sql/regress/seabase/TEST011 | 2 +
core/sql/regress/seabase/TEST031 | 74 ++++++
core/sql/regress/tools/runregr_privs1.ksh | 49 ++--
core/sql/regress/tools/runregr_privs2.ksh | 49 ++--
core/sql/sqlcomp/DefaultConstants.h | 7 +-
core/sql/sqlcomp/nadefaults.cpp | 42 +++-
.../TRAFODION/2.1/service_advisor.py | 3 +-
install/installer/traf_hortonworks_mods | 6 +
install/python-installer/configs/mod_cfgs.json | 1 +
31 files changed, 595 insertions(+), 134 deletions(-)
----------------------------------------------------------------------