You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by liuyu000 <gi...@git.apache.org> on 2017/09/10 11:37:10 UTC

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

GitHub user liuyu000 opened a pull request:

    https://github.com/apache/incubator-trafodion/pull/1229

    [Trafodion-2734] Create *Trafodion LOB Guide* and Add 1st Chapter *Introduction*

    Changes proposed in this pull request:
    1. Create **Trafodion LOB Guide**
    2. Add 1st Chapter **Introduction**

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/liuyu000/incubator-trafodion lob

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-trafodion/pull/1229.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1229
    
----
commit 4c54b742371206c67ac9bd564cc926f57f8ef01a
Author: liu.yu <yu...@esgyn.cn>
Date:   2017-09-10T11:26:08Z

    add lob folder and 1st chapter

----


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by sandhyasun <gi...@git.apache.org>.
Github user sandhyasun commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139335792
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    --- End diff --
    
    This restriction is correct. The source table that contains columns for an "external lob" only contains the name of the external lob file. Trafodion does not copy and store that data in our usual lob namespace - i.e user/trafodion/lobs. The descriptor files only contain the name of an hdfs file located somewhere in the system. When we do an insert-select from an external table, we copy the name of the "external" lob data file only.  So we need a place in the target table to store the external file name. This format can only be supported if the target table's column is also defined as external.  Today we allow insert-select from a hive varchar column, trafodion char or varchar column or a trafodion lob column that is not external. 


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772777
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    --- End diff --
    
    You've explained this very well.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773481
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +[#considerations]
    +==== Considerations
    +
    +If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.
    +
    +If the target file does not exist, you can create a target file by specifying the `OPTION`. 
    +
    +[#examples]
    +==== Examples
    +
    +* This example extracts LOB to the tlob130_txt1.txt:
    +
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
    +Success. Targetfile:tlob130_txt1.txt  Length: 19
    +```
    +
    +* This example extracts LOB to the tlob130_deep.jpg:
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"', 
    +Success. Targetfile:tlob130_deep.jpg  Length: 159018
    +```
    +
    +[#extract lob data into a user specified buffer]
    +=== Extract Lob Data into a User Specified Buffer
    +
    +Extract from a LOB column into a target user buffer of a specified size. Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until EOD. Then the extracted buffer data can be written to files or another location as the application chooses. 
    +
    +Extract LOB data into a user specified buffer like a cursor until EOD is returned. For this method, the user specifies an input buffer and specifies the input length in an in/out variable. 
    +
    +For each execution of the extract, Trafodion will return SUCCESS, ERROR or EOD. 
    --- End diff --
    
    Are SUCCESS, ERROR and EOD literals in some programming language (such as Java or C++)? Or is it a string that is returned by the Trafodion engine? Or something else? (The sqlci examples below don't mention SUCCESS, ERROR or EOD.) Seems like some additional explanation or examples are needed.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773014
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    --- End diff --
    
    Instead of "EOD", I'd suggest "end of data" (we haven't defined this acronym anywhere, and it's not clear that we need to).


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138542336
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    +
    +When a SQL table with LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138537922
  
    --- Diff: docs/lob_guide/pom.xml ---
    @@ -0,0 +1,301 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    + <!-- 
    +* @@@ START COPYRIGHT @@@                                                       
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +-->
    +  <modelVersion>4.0.0</modelVersion>
    +  <groupId>org.apache.trafodion</groupId>
    +  <artifactId>sql-reference-manual</artifactId>
    +  <version>${env.TRAFODION_VER}</version>
    +  <packaging>pom</packaging>
    +  <name>Trafodion SQL Reference Manual</name>
    +  <description>This manual describes reference information about the syntax of SQL statements, 
    --- End diff --
    
    Yes I've changed. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303453
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +[#considerations]
    +==== Considerations
    +
    +If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.
    +
    +If the target file does not exist, you can create a target file by specifying the `OPTION`. 
    +
    +[#examples]
    +==== Examples
    +
    +* This example extracts LOB to the tlob130_txt1.txt:
    +
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
    +Success. Targetfile:tlob130_txt1.txt  Length: 19
    +```
    +
    +* This example extracts LOB to the tlob130_deep.jpg:
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"', 
    +Success. Targetfile:tlob130_deep.jpg  Length: 159018
    +```
    +
    +[#extract lob data into a user specified buffer]
    +=== Extract Lob Data into a User Specified Buffer
    +
    +Extract from a LOB column into a target user buffer of a specified size. Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until EOD. Then the extracted buffer data can be written to files or another location as the application chooses. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303401
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138493805
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    --- End diff --
    
    In this section, are you trying to define the concept of "LOB" in general (that is, for any database), or just for Trafodion? Are you trying to define the notion of a LOB data type? Or a value that might be stored there?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139301423
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    --- End diff --
    
    Thanks Dave. :)
    
    I've changed it to:
    
    ...
    
    target-col-list-expr is: 
    target-colname[, target-colname]...
    
    target-colname is: 
    lob-column-name                  |
    other-column-name        
         
    ....


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138494095
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    --- End diff --
    
    "row store" is better ("row" is being used as an adjective)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138493175
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    --- End diff --
    
    "a LOB" would be better here. When LOB is being used as a singular noun, the indefinite article is called for.


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-trafodion/pull/1229


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138538583
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138768085
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is a lOB]
    +== What is a LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as a large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of row store and column store. 
    +
    +When a SQL table with a LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in an HDFS file while the Trafodion table contains a unique LOB handle for each LOB value. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    +
    +The naming conventions for the LOB tables are as follows: 
    +[caption="Table 1-2 "]
    +.Naming Conventions for LOB Tables
    +[cols="2*",options="header"]
    +|===
    +2+|*Naming Conventions*
    +| SQL Table                | TLOB 
    +| LOB MD table +
    +(one per SQL table containing LOB columns)
    +                           | LOBMD_{object UID}
    +| LOB Descriptor Handle Table +
    +(one per LOB column)
    +                           | LOBDescHandle_{object UID}_{LOB number}
    +| LOB Descriptor Chunks Table +
    +(one per LOB column)
    +                           | LOBDescChunks_{object UID}_{LOB number}
    +| LOB Data Table + 
    +(in HDFS one per LOB column)
    +                           | LOBP_{object UID}_{LOB number}
    +|===
    +
    +[#lob handle]
    +== LOB Handle
    +
    +The LOB handle is used to describe a LOB value. A SQL table that contains a LOB column will contain this handle structure in each row for each LOB value inserted.
    +
    +The actual LOB data will be stored in unstructured HDFS files as column store whereas the LOB handle—that describes the location, offset information, descriptor information and so on can be thought of as a unique identifier to the lob data, is stored in the database row of the table as a traditional row store. 
    +
    +The handle can be thought of as a file locator as is more commonly referred to in LOB related terminology. 
    +
    +[#external structure]
    +=== External Structure
    +
    +The following is an external structure that is stored in the row of a SQL table:
    +
    +LOBH00000002000100047335557604604880171074381106028370118212279894381354363017"TRAFODION"."SCH"
    +
    +[#lob restrictions]
    +== LOB Restrictions
    +
    +* LOB columns cannot appear in FROM clauses as join predicates.
    +* LOB columns cannot appear in STORE BY, ORDER BY or GROUP BY clauses.
    +* LOB columns cannot appear in WHERE clauses as predicates other than LIKE (meaning that no comparison is allowed).
    +* LOB columns cannot appear in SELECT clauses as aggregate function arguments.
    +* LOB columns cannot appear in SELECT DISTINCT clauses.
    +* LOB columns cannot be used as primary keys.
    +* LOB columns cannot be used in CREATE INDEX statements.
    +* LOB columns cannot be used in statistics update statements.
    +* The file that contains data to insert from or to extract to need to be on platform linux or HDFS files.
    --- End diff --
    
    Grammar:  "need to be" should be "needs to be" (because "needs" refers back to "The file" which is singular).


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139302837
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139053292
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    --- End diff --
    
    1. STORAGE clause 
    
    Syntax:
    ...
    lob-type is:
    BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    ...
    
    So the STORAGE clause is optional, it's ok if you don't specify it (the attachment
    ![create a sql table with lob columns](https://user-images.githubusercontent.com/20532956/30423142-8cfd079a-9974-11e7-9eb9-151183271ee5.jpg)
    ).
    
    2.  limit on LOB size
    I've asked Sandhya the some question, the answer from her:
    
    Minimum size:
    We allow for an “empty” LOB so technically that’s a  0 length lob. 
    We can insert using empty_blob(), empty_clob()  syntax.
    
    Maxmum size:
    Currently this is set to 10GB and it's controlled by the CQD LOB_MAX_SIZE. 
    But this can be changed depending on the customers needs to whatever they want. 
    There isn’t a real limit as such so we could go to 8 terabytes if needed but the user will have to configure enough HDFS disks to be able to support such a large LOB and several of them potentially. 


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139300972
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    --- End diff --
    
    Thanks Dave. :)
    
    I'm planning to change it to:
    
    insert-source is:     
    VALUES(column-expr[, column-expr]…)     
    
    column-expr is:
    lob-query-expr                        |
    other-query-expr                     
    
    ...
    
    Semantics
    •	other-query-expr  
    For more information about the other-query-expr, see the query-expr in  SELECT Statement.                      
       
    ...
    
    Any thoughs?



---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138493987
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    --- End diff --
    
    "... such as a large document or string" is better. Alternatively, "...such as large documents or strings"


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by sandhyasun <gi...@git.apache.org>.
Github user sandhyasun commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139335610
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    --- End diff --
    
    That's right Dave. This is for use by ODBC and JDBC driver. It cannot be done through sqlci. To test this, I wrote a C++ program that does CLI calls directly and mimics the calls from ODBC/JDBC . This is part of our regression suite.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772157
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    --- End diff --
    
    Perhaps this line should end with a colon, since the bullets in the list seem to be completing this sentence?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773196
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    --- End diff --
    
    Same comment as above


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138771564
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    --- End diff --
    
    Again, this syntax seems to suggest that a table containing LOB columns must have all LOB columns. But there will be a mixture of LOB and non-LOB columns in general.


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140177394
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    --- End diff --
    
    I'm planning to change the verb to gerund in all titles/headlines (for consistency) as below:
    
    Working with LOBs
    Creating a SQL Table with LOB Columns
    Updating a SQL Table Containing LOB Columns
    Selecting Column from a SQL Table Containing LOB Columns
    Extracting LOB Data from a SQL Table Containing LOB Columns
    ...
    
    Any comments?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138539084
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    +&#8226; 2.1  <<create a sql table with lob columns,Create a SQL Table with LOB Columns>> +
    +&#8226; 2.2  <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>> +
    +&#8226; 2.3  <<insert into a sql table containing lob columns using select clause,Insert into a SQL Table Containing LOB Columns Using Select Clause>> +
    +&#8226; 2.4  <<update a sql table containing lob columns,Update a SQL Table Containing LOB Columns>> +
    +&#8226; 2.5  <<select column from a sql table containing lob columns,Select Column from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.6  <<extract lob data from a sql table containing lob columns,Extract LOB Data from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.7  <<delete column from a sql table containing lob columns,Delete Column from a SQL Table Containing LOB columns>> +
    +&#8226; 2.8  <<drop a sql table containing lob columns,Drop a SQL Table Containing LOB Columns>> +
    +&#8226; 2.9  <<garbage collection,Garbage Collection>> +
    +&#8226; 2.10 <<cleanup of a sql table containing lob columns,Cleanup of a SQL Table Containing LOB Columns>> +
    +&#8226; 2.11 <<showddl for lob,SHOWDDL for LOB>> +
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140176015
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    --- End diff --
    
    Thanks Dave! :)
    OK, I've removed them.


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139533707
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    --- End diff --
    
    Maybe it is the periods at the end of the bullet items that seem odd to me. The bullets aren't full sentences in this example. Perhaps we should just remove the periods on the bullets?


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139532497
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    --- End diff --
    
    I think this is fine.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138539061
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138796677
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    --- End diff --
    
    Thanks Dave.
    
    Using gerund or verb (base) in title/headline is a style issue that should be followed consistently.
    
    Gerundive (continuous) form is common in title and verb form seems more imperative, but given that 
    1. the topic title plays a huge role in the search results
    2. Trafodion has many users of non-native English speakers
    
    I think the verb form might be better as it more likely to match what people will search for, and uses the more familiar form of the verb. 
    
    Any thoughts?


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140945525
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +[#considerations]
    +==== Considerations
    +
    +If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.
    +
    +If the target file does not exist, you can create a target file by specifying the `OPTION`. 
    +
    +[#examples]
    +==== Examples
    +
    +* This example extracts LOB to the tlob130_txt1.txt:
    +
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
    +Success. Targetfile:tlob130_txt1.txt  Length: 19
    +```
    +
    +* This example extracts LOB to the tlob130_deep.jpg:
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"', 
    +Success. Targetfile:tlob130_deep.jpg  Length: 159018
    +```
    +
    +[#extract lob data into a user specified buffer]
    +=== Extract Lob Data into a User Specified Buffer
    +
    +Extract from a LOB column into a target user buffer of a specified size. Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until EOD. Then the extracted buffer data can be written to files or another location as the application chooses. 
    +
    +Extract LOB data into a user specified buffer like a cursor until EOD is returned. For this method, the user specifies an input buffer and specifies the input length in an in/out variable. 
    +
    +For each execution of the extract, Trafodion will return SUCCESS, ERROR or EOD. 
    --- End diff --
    
    Suggestion from Sandhya:
    
    Change 
    "For each execution of the extract, Trafodion will return SUCCESS, ERROR or EOD. 
    For the cases of SUCCESS or EOD, a length will also be returned to the user, so the user knows exactly how much data was actually extracted and returned. "
    
    to
    
    "Each successful execution of the extract will also return the length in the parameter specified by the caller, so the user knows how much data was actually extracted and returned. "



---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139533191
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    --- End diff --
    
    Looks good to me.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138537372
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    --- End diff --
    
    Thanks Dave, I've corrected and added. 


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138769430
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    --- End diff --
    
    This syntax suggests that when you have lob columns in a table, you can't have other columns as well. Perhaps something like "CREATE TABLE table-name ( column_spec [, column_spec] ... [, primary_key_spec] )", then "column_spec is: lob-column-spec | other column spec"



---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139078004
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    --- End diff --
    
    Thanks Dave.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138768237
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is a lOB]
    +== What is a LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as a large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of row store and column store. 
    +
    +When a SQL table with a LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in an HDFS file while the Trafodion table contains a unique LOB handle for each LOB value. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    +
    +The naming conventions for the LOB tables are as follows: 
    +[caption="Table 1-2 "]
    +.Naming Conventions for LOB Tables
    +[cols="2*",options="header"]
    +|===
    +2+|*Naming Conventions*
    +| SQL Table                | TLOB 
    +| LOB MD table +
    +(one per SQL table containing LOB columns)
    +                           | LOBMD_{object UID}
    +| LOB Descriptor Handle Table +
    +(one per LOB column)
    +                           | LOBDescHandle_{object UID}_{LOB number}
    +| LOB Descriptor Chunks Table +
    +(one per LOB column)
    +                           | LOBDescChunks_{object UID}_{LOB number}
    +| LOB Data Table + 
    +(in HDFS one per LOB column)
    +                           | LOBP_{object UID}_{LOB number}
    +|===
    +
    +[#lob handle]
    +== LOB Handle
    +
    +The LOB handle is used to describe a LOB value. A SQL table that contains a LOB column will contain this handle structure in each row for each LOB value inserted.
    +
    +The actual LOB data will be stored in unstructured HDFS files as column store whereas the LOB handle—that describes the location, offset information, descriptor information and so on can be thought of as a unique identifier to the lob data, is stored in the database row of the table as a traditional row store. 
    +
    +The handle can be thought of as a file locator as is more commonly referred to in LOB related terminology. 
    +
    +[#external structure]
    +=== External Structure
    +
    +The following is an external structure that is stored in the row of a SQL table:
    +
    +LOBH00000002000100047335557604604880171074381106028370118212279894381354363017"TRAFODION"."SCH"
    +
    +[#lob restrictions]
    +== LOB Restrictions
    +
    +* LOB columns cannot appear in FROM clauses as join predicates.
    +* LOB columns cannot appear in STORE BY, ORDER BY or GROUP BY clauses.
    +* LOB columns cannot appear in WHERE clauses as predicates other than LIKE (meaning that no comparison is allowed).
    +* LOB columns cannot appear in SELECT clauses as aggregate function arguments.
    +* LOB columns cannot appear in SELECT DISTINCT clauses.
    +* LOB columns cannot be used as primary keys.
    +* LOB columns cannot be used in CREATE INDEX statements.
    +* LOB columns cannot be used in statistics update statements.
    +* The file that contains data to insert from or to extract to need to be on platform linux or HDFS files.
    --- End diff --
    
    Also, "on platform linux or HDFS files." I think you mean, "on platform as a Linux or HDFS file."


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138769506
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    --- End diff --
    
    Grammar. Should be "... supports ..."


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138557787
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    +
    +When a SQL table with LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in HDFS while the Trafodion table contains a unique LOB handle for each row/column. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)
    
    Answer from Sandhya:
    Each lob column in any table will have  it’s own unique lob data file. So any lob data belonging to that column will be stored in that HDFS file. 
    So if a table has 3 columns, you will see 3 lob data files with the first part of the name being the same and the column number will be appended to it.
    
    For example, for a table with 2 lob columns:
    Location: /user/trafodion/lobs
    DataFile: LOBP_04474425229029907479_0001
    
    Location: /user/trafodion/lobs
    DataFile: LOBP_04474425229029907479_0002


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138767875
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is a lOB]
    +== What is a LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as a large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of row store and column store. 
    +
    +When a SQL table with a LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in an HDFS file while the Trafodion table contains a unique LOB handle for each LOB value. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    +
    +The naming conventions for the LOB tables are as follows: 
    +[caption="Table 1-2 "]
    +.Naming Conventions for LOB Tables
    +[cols="2*",options="header"]
    +|===
    +2+|*Naming Conventions*
    +| SQL Table                | TLOB 
    +| LOB MD table +
    +(one per SQL table containing LOB columns)
    +                           | LOBMD_{object UID}
    +| LOB Descriptor Handle Table +
    +(one per LOB column)
    +                           | LOBDescHandle_{object UID}_{LOB number}
    +| LOB Descriptor Chunks Table +
    +(one per LOB column)
    +                           | LOBDescChunks_{object UID}_{LOB number}
    +| LOB Data Table + 
    +(in HDFS one per LOB column)
    +                           | LOBP_{object UID}_{LOB number}
    +|===
    +
    +[#lob handle]
    +== LOB Handle
    +
    +The LOB handle is used to describe a LOB value. A SQL table that contains a LOB column will contain this handle structure in each row for each LOB value inserted.
    +
    +The actual LOB data will be stored in unstructured HDFS files as column store whereas the LOB handle—that describes the location, offset information, descriptor information and so on can be thought of as a unique identifier to the lob data, is stored in the database row of the table as a traditional row store. 
    +
    +The handle can be thought of as a file locator as is more commonly referred to in LOB related terminology. 
    +
    +[#external structure]
    +=== External Structure
    +
    +The following is an external structure that is stored in the row of a SQL table:
    --- End diff --
    
    Do you mean, "The following is an example of an external structure..."?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138771906
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    --- End diff --
    
    The last sentence surprises me actually. I would have thought that INSERT/SELECT might be a way to copy LOBs from external files into internal LOB values. I'm not saying this is incorrect, though (I don't know if it is). You may want to check on this.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773309
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +[#considerations]
    +==== Considerations
    +
    +If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.
    +
    +If the target file does not exist, you can create a target file by specifying the `OPTION`. 
    +
    +[#examples]
    +==== Examples
    +
    +* This example extracts LOB to the tlob130_txt1.txt:
    +
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
    +Success. Targetfile:tlob130_txt1.txt  Length: 19
    +```
    +
    +* This example extracts LOB to the tlob130_deep.jpg:
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"', 
    +Success. Targetfile:tlob130_deep.jpg  Length: 159018
    +```
    +
    +[#extract lob data into a user specified buffer]
    +=== Extract Lob Data into a User Specified Buffer
    +
    +Extract from a LOB column into a target user buffer of a specified size. Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until EOD. Then the extracted buffer data can be written to files or another location as the application chooses. 
    --- End diff --
    
    Suggest "The Trafodion engine..."


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138492480
  
    --- Diff: docs/lob_guide/pom.xml ---
    @@ -0,0 +1,301 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    + <!-- 
    +* @@@ START COPYRIGHT @@@                                                       
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +-->
    +  <modelVersion>4.0.0</modelVersion>
    +  <groupId>org.apache.trafodion</groupId>
    +  <artifactId>sql-reference-manual</artifactId>
    +  <version>${env.TRAFODION_VER}</version>
    +  <packaging>pom</packaging>
    +  <name>Trafodion SQL Reference Manual</name>
    --- End diff --
    
    Shouldn't this be different? Trafodion SQL Large Objects Guide perhaps?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139076734
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    --- End diff --
    
    Thanks Dave, for your eagle eyes.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138770951
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    --- End diff --
    
    Grammar: "is simple string data" sounds better to my ears. (The reason is that "data" is plural. Though increasingly "data" in everyday English is often construed as singular. But my ears are old.)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139076558
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    --- End diff --
    
    Thanks Dave.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138538389
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139533296
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    --- End diff --
    
    Looks good.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139301063
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    --- End diff --
    
    Thanks Dave.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139302735
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138845773
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    --- End diff --
    
    Thanks Dave! :)
    
    I'm planning to change it to:
    
    CREATE TABLE table-name (column-spec[, column-spec]…)
    
    column-spec is:
    lob-column-spec          | 
    other-column-spec
    
    lob-column-spec is:
    column-name {lob-data-type} [column-constraint]
    
    other-column-spec is:
    column-name {data-type} [column-constraint]
    
    lob-type is:
    BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    
    ...
    
    Any comments?


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140444657
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303423
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138771380
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    --- End diff --
    
    I'm trying to imagine when this makes sense. I suppose it might make sense in an ODBC or JDBC application where the code that is generating the INSERT statement knows about in-memory buffer addresses. It probably doesn't make sense in sqlci or trafci. Is that your understanding also?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138771650
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    --- End diff --
    
    "Names" should be "names" (lower case) since it isn't starting the sentence.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138538113
  
    --- Diff: docs/lob_guide/pom.xml ---
    @@ -0,0 +1,301 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    + <!-- 
    +* @@@ START COPYRIGHT @@@                                                       
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +-->
    +  <modelVersion>4.0.0</modelVersion>
    +  <groupId>org.apache.trafodion</groupId>
    +  <artifactId>sql-reference-manual</artifactId>
    +  <version>${env.TRAFODION_VER}</version>
    +  <packaging>pom</packaging>
    +  <name>Trafodion SQL Reference Manual</name>
    +  <description>This manual describes reference information about the syntax of SQL statements, 
    --- End diff --
    
    Thanks Dave.
    Yes I've changed. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139301178
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138770192
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    --- End diff --
    
    The space after "lobs" should be removed


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138542183
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139302665
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139302827
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    --- End diff --
    
    Sorry I don't quite understand. Do you mean put a colon after "a literal needs to be provided"?
    
    ![update using parameters or functions](https://user-images.githubusercontent.com/20532956/30518514-85c18a4c-9bb2-11e7-842d-0b2aacf27527.jpg)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139302835
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138494272
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    +
    +When a SQL table with LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in HDFS while the Trafodion table contains a unique LOB handle for each row/column. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    --- End diff --
    
    "an HDFS file" is better. A question: Is each LOB value stored in a separate HDFS file? Or are several LOB values stored in one HDFS file? Also, "...a unique LOB handle for each row/column" might be better stated, "...a unique LOB handle for each LOB value".  


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138540283
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    +&#8226; 2.1  <<create a sql table with lob columns,Create a SQL Table with LOB Columns>> +
    +&#8226; 2.2  <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>> +
    +&#8226; 2.3  <<insert into a sql table containing lob columns using select clause,Insert into a SQL Table Containing LOB Columns Using Select Clause>> +
    +&#8226; 2.4  <<update a sql table containing lob columns,Update a SQL Table Containing LOB Columns>> +
    +&#8226; 2.5  <<select column from a sql table containing lob columns,Select Column from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.6  <<extract lob data from a sql table containing lob columns,Extract LOB Data from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.7  <<delete column from a sql table containing lob columns,Delete Column from a SQL Table Containing LOB columns>> +
    +&#8226; 2.8  <<drop a sql table containing lob columns,Drop a SQL Table Containing LOB Columns>> +
    +&#8226; 2.9  <<garbage collection,Garbage Collection>> +
    +&#8226; 2.10 <<cleanup of a sql table containing lob columns,Cleanup of a SQL Table Containing LOB Columns>> +
    +&#8226; 2.11 <<showddl for lob,SHOWDDL for LOB>> +
    +&#8226; 2.12 <<get command for lob tables,Get Command for LOB Tables>>
    +|===
    +
    +== Notation Conventions
    +This list summarizes the notation conventions for syntax presentation in this manual.
    +
    +* UPPERCASE LETTERS
    ++
    +Uppercase letters indicate keywords and reserved words. Type these items exactly as shown. Items not enclosed in brackets are required. 
    ++
    +```
    +SELECT
    +```
    +
    +* lowercase letters
    ++
    +Lowercase letters, regardless of font, indicate variable items that you supply. Items not enclosed in brackets are required.
    ++
    +```
    +file-name
    +```
    +
    +* &#91; &#93; Brackets 
    ++
    +Brackets enclose optional syntax items.
    ++
    +```
    +DATETIME [start-field TO] end-field
    +```
    ++
    +A group of items enclosed in brackets is a list from which you can choose one item or none.
    ++
    +The items in the list can be arranged either vertically, with aligned brackets on each side of the list, or horizontally, enclosed in a pair of brackets and separated by vertical lines.
    ++
    +For example: 
    ++
    +```
    +DROP SCHEMA schema [CASCADE]
    +DROP SCHEMA schema [ CASCADE | RESTRICT ]
    +```
    +
    +<<<
    +* { } Braces 
    ++
    +Braces enclose required syntax items.
    ++
    +```
    +FROM { grantee [, grantee ] ... }
    +```
    ++ 
    +A group of items enclosed in braces is a list from which you are required to choose one item.
    ++
    +The items in the list can be arranged either vertically, with aligned braces on each side of the list, or horizontally, enclosed in a pair of braces and separated by vertical lines.
    ++
    +For example:
    ++
    +```
    +INTERVAL { start-field TO end-field }
    +{ single-field } 
    +INTERVAL { start-field TO end-field | single-field }
    +``` 
    +* | Vertical Line 
    ++
    +A vertical line separates alternatives in a horizontal list that is enclosed in brackets or braces.
    ++
    +```
    +{expression | NULL} 
    +```
    +* &#8230; Ellipsis
    ++
    +An ellipsis immediately following a pair of brackets or braces indicates that you can repeat the enclosed sequence of syntax items any number of times.
    ++
    +```
    +ATTRIBUTE[S] attribute [, attribute] ...
    +{, sql-expression } ...
    +```
    ++ 
    +An ellipsis immediately following a single syntax item indicates that you can repeat that syntax item any number of times.
    ++
    +For example:
    ++
    +```
    +expression-n ...
    +```
    +
    +* Punctuation
    ++
    +Parentheses, commas, semicolons, and other symbols not previously described must be typed as shown.
    ++
    +```
    +DAY (datetime-expression)
    +@script-file 
    +```
    ++
    +Quotation marks around a symbol such as a bracket or brace indicate the symbol is a required character that you must type as shown.
    ++
    +For example:
    ++
    +```
    +"{" module-name [, module-name] ... "}"
    +```
    +
    +* Item Spacing
    ++
    +Spaces shown between items are required unless one of the items is a punctuation symbol such as a parenthesis or a comma.
    ++
    +```
    +DAY (datetime-expression) DAY(datetime-expression)
    +```
    ++
    +If there is no space between two items, spaces are not permitted. In this example, no spaces are permitted between the period and any other items:
    ++
    +```
    +myfile.sh
    +```
    +
    +* Line Spacing
    ++
    +If the syntax of a command is too long to fit on a single line, each continuation line is indented three spaces and is separated from the preceding line by a blank line.
    ++
    +This spacing distinguishes items in a continuation line from items in a vertical list of selections. 
    ++
    +```
    +match-value [NOT] LIKE _pattern
    +   [ESCAPE esc-char-expression] 
    +```
    +
    +== Comments Encouraged
    +We encourage your comments concerning this document. We are committed to providing documentation that meets your
    +needs. Send any errors found, suggestions for improvement, or compliments to user@trafodion.incubator.apache.org.
    --- End diff --
    
    Yes, hope we could get more and more encouragements. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303398
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    --- End diff --
    
    Same as the previous reply: "Using gerund or verb (base) in title/headline is a style issue that should be followed consistently....".


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772431
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    --- End diff --
    
    Same here


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139533345
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    --- End diff --
    
    Thanks!


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140364441
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    --- End diff --
    
    I love it :-)


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139535246
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    --- End diff --
    
    It seems ungrammatical to my ear. An imperative tense would clearly not be correct. We aren't telling the reader to "Work with LOBs!" (note the plural would be required, or alternatively the indefinite article, "Work with a LOB!"). Rather we are trying to describe how one works with LOBs. To me using the gerund seems most natural (in all cases, not just here); I would recommend changing the style to do that. Another possibility is to use the infinitive, e.g. "How to work with LOBs" (note again the plural). That might be more natural for English-as-a-second-language users whose native language lacks gerund-like inflections.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138768441
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    --- End diff --
    
    Consider "Working with LOBs"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138492812
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    --- End diff --
    
    Typo, "dwscribes". Also, might want to say "...how to use Large Object (LOB) datatypes in Trafodion SQL."


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138492937
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    --- End diff --
    
    Grammar: "Introduces LOBs ..." is better.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138542281
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    --- End diff --
    
    Thanks Dave.
    I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772478
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    --- End diff --
    
    And here, and in the examples below too


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772402
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    --- End diff --
    
    Instead of "the c2", I suggest "the value of c2"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138541983
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    --- End diff --
    
    1. Just for Trafodion.
    2. Both. Define the notion first, and then the value (1.4 LOB Handle).


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140161665
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    --- End diff --
    
    Thanks for the detailed explanation! :)
    I've changed to "How to work with LOBs".


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140168341
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    --- End diff --
    
    Thanks Dave! :)
    I've corrected.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303434
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138493542
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    +&#8226; 2.1  <<create a sql table with lob columns,Create a SQL Table with LOB Columns>> +
    +&#8226; 2.2  <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>> +
    +&#8226; 2.3  <<insert into a sql table containing lob columns using select clause,Insert into a SQL Table Containing LOB Columns Using Select Clause>> +
    +&#8226; 2.4  <<update a sql table containing lob columns,Update a SQL Table Containing LOB Columns>> +
    +&#8226; 2.5  <<select column from a sql table containing lob columns,Select Column from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.6  <<extract lob data from a sql table containing lob columns,Extract LOB Data from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.7  <<delete column from a sql table containing lob columns,Delete Column from a SQL Table Containing LOB columns>> +
    +&#8226; 2.8  <<drop a sql table containing lob columns,Drop a SQL Table Containing LOB Columns>> +
    +&#8226; 2.9  <<garbage collection,Garbage Collection>> +
    +&#8226; 2.10 <<cleanup of a sql table containing lob columns,Cleanup of a SQL Table Containing LOB Columns>> +
    +&#8226; 2.11 <<showddl for lob,SHOWDDL for LOB>> +
    +&#8226; 2.12 <<get command for lob tables,Get Command for LOB Tables>>
    +|===
    +
    +== Notation Conventions
    +This list summarizes the notation conventions for syntax presentation in this manual.
    +
    +* UPPERCASE LETTERS
    ++
    +Uppercase letters indicate keywords and reserved words. Type these items exactly as shown. Items not enclosed in brackets are required. 
    ++
    +```
    +SELECT
    +```
    +
    +* lowercase letters
    ++
    +Lowercase letters, regardless of font, indicate variable items that you supply. Items not enclosed in brackets are required.
    ++
    +```
    +file-name
    +```
    +
    +* &#91; &#93; Brackets 
    ++
    +Brackets enclose optional syntax items.
    ++
    +```
    +DATETIME [start-field TO] end-field
    +```
    ++
    +A group of items enclosed in brackets is a list from which you can choose one item or none.
    ++
    +The items in the list can be arranged either vertically, with aligned brackets on each side of the list, or horizontally, enclosed in a pair of brackets and separated by vertical lines.
    ++
    +For example: 
    ++
    +```
    +DROP SCHEMA schema [CASCADE]
    +DROP SCHEMA schema [ CASCADE | RESTRICT ]
    +```
    +
    +<<<
    +* { } Braces 
    ++
    +Braces enclose required syntax items.
    ++
    +```
    +FROM { grantee [, grantee ] ... }
    +```
    ++ 
    +A group of items enclosed in braces is a list from which you are required to choose one item.
    ++
    +The items in the list can be arranged either vertically, with aligned braces on each side of the list, or horizontally, enclosed in a pair of braces and separated by vertical lines.
    ++
    +For example:
    ++
    +```
    +INTERVAL { start-field TO end-field }
    +{ single-field } 
    +INTERVAL { start-field TO end-field | single-field }
    +``` 
    +* | Vertical Line 
    ++
    +A vertical line separates alternatives in a horizontal list that is enclosed in brackets or braces.
    ++
    +```
    +{expression | NULL} 
    +```
    +* &#8230; Ellipsis
    ++
    +An ellipsis immediately following a pair of brackets or braces indicates that you can repeat the enclosed sequence of syntax items any number of times.
    ++
    +```
    +ATTRIBUTE[S] attribute [, attribute] ...
    +{, sql-expression } ...
    +```
    ++ 
    +An ellipsis immediately following a single syntax item indicates that you can repeat that syntax item any number of times.
    ++
    +For example:
    ++
    +```
    +expression-n ...
    +```
    +
    +* Punctuation
    ++
    +Parentheses, commas, semicolons, and other symbols not previously described must be typed as shown.
    ++
    +```
    +DAY (datetime-expression)
    +@script-file 
    +```
    ++
    +Quotation marks around a symbol such as a bracket or brace indicate the symbol is a required character that you must type as shown.
    ++
    +For example:
    ++
    +```
    +"{" module-name [, module-name] ... "}"
    +```
    +
    +* Item Spacing
    ++
    +Spaces shown between items are required unless one of the items is a punctuation symbol such as a parenthesis or a comma.
    ++
    +```
    +DAY (datetime-expression) DAY(datetime-expression)
    +```
    ++
    +If there is no space between two items, spaces are not permitted. In this example, no spaces are permitted between the period and any other items:
    ++
    +```
    +myfile.sh
    +```
    +
    +* Line Spacing
    ++
    +If the syntax of a command is too long to fit on a single line, each continuation line is indented three spaces and is separated from the preceding line by a blank line.
    ++
    +This spacing distinguishes items in a continuation line from items in a vertical list of selections. 
    ++
    +```
    +match-value [NOT] LIKE _pattern
    +   [ESCAPE esc-char-expression] 
    +```
    +
    +== Comments Encouraged
    +We encourage your comments concerning this document. We are committed to providing documentation that meets your
    +needs. Send any errors found, suggestions for improvement, or compliments to user@trafodion.incubator.apache.org.
    --- End diff --
    
    Yes, especially compliments :-)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139303375
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    --- End diff --
    
    Thanks Dave. :)


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139533238
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    --- End diff --
    
    Thanks!


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773158
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    --- End diff --
    
    Suggest "If the target file does not exist, Trafodion will create it and write to it." (I assume Trafodion is creating the file here and not creating a table somewhere.)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138492594
  
    --- Diff: docs/lob_guide/pom.xml ---
    @@ -0,0 +1,301 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    + <!-- 
    +* @@@ START COPYRIGHT @@@                                                       
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +-->
    +  <modelVersion>4.0.0</modelVersion>
    +  <groupId>org.apache.trafodion</groupId>
    +  <artifactId>sql-reference-manual</artifactId>
    +  <version>${env.TRAFODION_VER}</version>
    +  <packaging>pom</packaging>
    +  <name>Trafodion SQL Reference Manual</name>
    +  <description>This manual describes reference information about the syntax of SQL statements, 
    --- End diff --
    
    The description probably should be changed as well.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138493454
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    +&#8226; 1.2 <<lob data types,LOB Data Types>> +
    +&#8226; 1.3 <<lob storage,LOB Storage>> +
    +&#8226; 1.4 <<lob handle,LOB Handle>> +
    +&#8226; 1.5 <<lob restrictions,LOB Restrictions>> +
    +&#8226; 1.6 <<lob related sql statements and functions,LOB Related SQL Statements and Functions>>.
    +
    +| <<Work with LOB,Work with LOB>>              | Explains how to use LOB with SQL statement and covers following topics: +
    +&#8226; 2.1  <<create a sql table with lob columns,Create a SQL Table with LOB Columns>> +
    +&#8226; 2.2  <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>> +
    +&#8226; 2.3  <<insert into a sql table containing lob columns using select clause,Insert into a SQL Table Containing LOB Columns Using Select Clause>> +
    +&#8226; 2.4  <<update a sql table containing lob columns,Update a SQL Table Containing LOB Columns>> +
    +&#8226; 2.5  <<select column from a sql table containing lob columns,Select Column from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.6  <<extract lob data from a sql table containing lob columns,Extract LOB Data from a SQL Table Containing LOB Columns>> +
    +&#8226; 2.7  <<delete column from a sql table containing lob columns,Delete Column from a SQL Table Containing LOB columns>> +
    +&#8226; 2.8  <<drop a sql table containing lob columns,Drop a SQL Table Containing LOB Columns>> +
    +&#8226; 2.9  <<garbage collection,Garbage Collection>> +
    +&#8226; 2.10 <<cleanup of a sql table containing lob columns,Cleanup of a SQL Table Containing LOB Columns>> +
    +&#8226; 2.11 <<showddl for lob,SHOWDDL for LOB>> +
    --- End diff --
    
    "for LOBs" might be better. Alternatively, "SHOWDDL for LOB columns"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138494171
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is lOB]
    +== What is LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of rows store and column store. 
    +
    +When a SQL table with LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    --- End diff --
    
    "with a LOB column" is better


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138769886
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    --- End diff --
    
    By "the Object UID" I assume we mean "the table's Object UID"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138770236
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    --- End diff --
    
    Do you mean, "As rows are added to this table, ..."?


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138773336
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    +
    +* Extract lob data into a file for a given lob handle
    ++
    +Extract from a LOB column straight into an on-platform linux or HDFS file in one shot. 
    +
    ++
    +Trafodion engine will take care of buffering the data and retrieve the lob data into the target file.
    +
    +* Extract lob data into a user specified buffer
    ++
    +Extract from a LOB column into a target user buffer of a specified size. 
    ++
    +Trafodion engine will retrieve exactly the amount of requested data. 
    ++
    +The user/application will be responsible for redriving the extract until EOD. 
    ++
    +The extracted buffer data can then be written to files or another location as the application chooses. 
    +
    +[#extract lob data into a file for a given lob handle]
    +=== Extract Lob Data into a File for a Given Lob Handle 
    +
    +[#syntax]
    +==== Syntax
    +
    +```
    +EXTRACT LOBTOFILE (LOB 'lob handle as quoted string', 'filename URI format' [OPTION]) 
    +```
    +
    +```
    +OPTION is:                                         
    +[, TRUNCATE]
    +[, CREATE, TRUNCATE]
    +[, APPEND]   
    +[, CREATE, APPEND]
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +* `TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it.
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, TRUNCATE`
    ++
    +If the target file exists, Trafodion will truncate and write to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +* `APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, an error will be raised.
    +
    +* `CREATE, APPEND`
    ++
    +If the target file exists, Trafodion will append to it. 
    ++
    +If the target file does not exist, create a table, then write to it. 
    +
    +[#considerations]
    +==== Considerations
    +
    +If the target file exists, the `OPTION` must be specified, or else an error will be raised. This is the default behavior.
    +
    +If the target file does not exist, you can create a target file by specifying the `OPTION`. 
    +
    +[#examples]
    +==== Examples
    +
    +* This example extracts LOB to the tlob130_txt1.txt:
    +
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583625519884121437206093184618212317486018305654020"TRAFODION"."LOB130"', 'tlob130_txt1.txt');
    +Success. Targetfile:tlob130_txt1.txt  Length: 19
    +```
    +
    +* This example extracts LOB to the tlob130_deep.jpg:
    ++
    +```
    +EXTRACT LOBTOFILE (LOB 'LOBH00000200010520117997292583681719884121437210516812518212317486062586654020"TRAFODION"."LOB130"', 
    +Success. Targetfile:tlob130_deep.jpg  Length: 159018
    +```
    +
    +[#extract lob data into a user specified buffer]
    +=== Extract Lob Data into a User Specified Buffer
    +
    +Extract from a LOB column into a target user buffer of a specified size. Trafodion engine will retrieve exactly the amount of requested data. The user/application will be responsible for redriving the extract until EOD. Then the extracted buffer data can be written to files or another location as the application chooses. 
    --- End diff --
    
    Suggest also "end of data" instead of EOD.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r139075962
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    --- End diff --
    
    Thanks Dave, I've corrected. :)



---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138538076
  
    --- Diff: docs/lob_guide/pom.xml ---
    @@ -0,0 +1,301 @@
    +<?xml version="1.0" encoding="UTF-8"?>
    +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    +  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    + <!-- 
    +* @@@ START COPYRIGHT @@@                                                       
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +-->
    +  <modelVersion>4.0.0</modelVersion>
    +  <groupId>org.apache.trafodion</groupId>
    +  <artifactId>sql-reference-manual</artifactId>
    +  <version>${env.TRAFODION_VER}</version>
    +  <packaging>pom</packaging>
    +  <name>Trafodion SQL Reference Manual</name>
    --- End diff --
    
    Thanks for your reminding. :) 
    Yes I've changed it to Trafodion SQL Large Objects Guide.


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140166131
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    --- End diff --
    
    Thanks Dave! :)
    I've corrected.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138785363
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is a lOB]
    +== What is a LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as a large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of row store and column store. 
    +
    +When a SQL table with a LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in an HDFS file while the Trafodion table contains a unique LOB handle for each LOB value. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    +
    +The naming conventions for the LOB tables are as follows: 
    +[caption="Table 1-2 "]
    +.Naming Conventions for LOB Tables
    +[cols="2*",options="header"]
    +|===
    +2+|*Naming Conventions*
    +| SQL Table                | TLOB 
    +| LOB MD table +
    +(one per SQL table containing LOB columns)
    +                           | LOBMD_{object UID}
    +| LOB Descriptor Handle Table +
    +(one per LOB column)
    +                           | LOBDescHandle_{object UID}_{LOB number}
    +| LOB Descriptor Chunks Table +
    +(one per LOB column)
    +                           | LOBDescChunks_{object UID}_{LOB number}
    +| LOB Data Table + 
    +(in HDFS one per LOB column)
    +                           | LOBP_{object UID}_{LOB number}
    +|===
    +
    +[#lob handle]
    +== LOB Handle
    +
    +The LOB handle is used to describe a LOB value. A SQL table that contains a LOB column will contain this handle structure in each row for each LOB value inserted.
    +
    +The actual LOB data will be stored in unstructured HDFS files as column store whereas the LOB handle—that describes the location, offset information, descriptor information and so on can be thought of as a unique identifier to the lob data, is stored in the database row of the table as a traditional row store. 
    +
    +The handle can be thought of as a file locator as is more commonly referred to in LOB related terminology. 
    +
    +[#external structure]
    +=== External Structure
    +
    +The following is an external structure that is stored in the row of a SQL table:
    --- End diff --
    
    Thanks Dave.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138771169
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    --- End diff --
    
    Suggest "value" instead of "column/row"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138785512
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/introduction.adoc ---
    @@ -0,0 +1,152 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#introduction]
    += Introduction
    +
    +[#what is a lOB]
    +== What is a LOB
    +
    +LOB (Large Object), a set of large object data types used to store large volumes of data, provides random and piece-wise access to the data. 
    +
    +Database support for LOB is not universal.
    +
    +[#lob data types]
    +== LOB Data Types
    +
    +The following are the data types supported by Trafodion. +
    +
    +
    +[caption="Table 1-1 "]
    +.Descriptions for BLOB and CLOB
    +[cols="30%,70%",options="header"]
    +|===
    +| *Data Type*         | *Desciprtion*
    +| BLOB                | Binary Large Object. + 
    +
    +Holds large blocks of unstructured data in binary format. +
    +
    +BLOB is ideal for storing and handling unstructured data, such as images, audio, video and other multimedia objects.
    +| CLOB                | Character Large Object. +
    +
    +Stores unusually large amounts of string data in character set format. +
    +
    +CLOB is ideal for storing and processing semi-structured data, such as a large document or string.
    +|===
    +
    +[#lob storage]
    +== LOB Storage
    +
    +The basic design idea is to allow a database row to have multiple LOB columns and a table to have multiple such rows. The design takes a combination approach of row store and column store. 
    +
    +When a SQL table with a LOB column is created, there are several dependent objects that are created to hold metadata information on the LOB as well as the data.
    +
    +The LOB data itself is stored in an HDFS file while the Trafodion table contains a unique LOB handle for each LOB value. The LOB handle provides the identifier that is used to query the LOB metadata tables to find the location of the LOB data files.
    +
    +The naming conventions for the LOB tables are as follows: 
    +[caption="Table 1-2 "]
    +.Naming Conventions for LOB Tables
    +[cols="2*",options="header"]
    +|===
    +2+|*Naming Conventions*
    +| SQL Table                | TLOB 
    +| LOB MD table +
    +(one per SQL table containing LOB columns)
    +                           | LOBMD_{object UID}
    +| LOB Descriptor Handle Table +
    +(one per LOB column)
    +                           | LOBDescHandle_{object UID}_{LOB number}
    +| LOB Descriptor Chunks Table +
    +(one per LOB column)
    +                           | LOBDescChunks_{object UID}_{LOB number}
    +| LOB Data Table + 
    +(in HDFS one per LOB column)
    +                           | LOBP_{object UID}_{LOB number}
    +|===
    +
    +[#lob handle]
    +== LOB Handle
    +
    +The LOB handle is used to describe a LOB value. A SQL table that contains a LOB column will contain this handle structure in each row for each LOB value inserted.
    +
    +The actual LOB data will be stored in unstructured HDFS files as column store whereas the LOB handle—that describes the location, offset information, descriptor information and so on can be thought of as a unique identifier to the lob data, is stored in the database row of the table as a traditional row store. 
    +
    +The handle can be thought of as a file locator as is more commonly referred to in LOB related terminology. 
    +
    +[#external structure]
    +=== External Structure
    +
    +The following is an external structure that is stored in the row of a SQL table:
    +
    +LOBH00000002000100047335557604604880171074381106028370118212279894381354363017"TRAFODION"."SCH"
    +
    +[#lob restrictions]
    +== LOB Restrictions
    +
    +* LOB columns cannot appear in FROM clauses as join predicates.
    +* LOB columns cannot appear in STORE BY, ORDER BY or GROUP BY clauses.
    +* LOB columns cannot appear in WHERE clauses as predicates other than LIKE (meaning that no comparison is allowed).
    +* LOB columns cannot appear in SELECT clauses as aggregate function arguments.
    +* LOB columns cannot appear in SELECT DISTINCT clauses.
    +* LOB columns cannot be used as primary keys.
    +* LOB columns cannot be used in CREATE INDEX statements.
    +* LOB columns cannot be used in statistics update statements.
    +* The file that contains data to insert from or to extract to need to be on platform linux or HDFS files.
    --- End diff --
    
    Thanks Dave.
    Yes, I've corrected. :)


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138772879
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    +
    +target-colname is: 
    +trafodion lob column name
    +
    +source-col-list-expr is: 
    +source-colname[, source-colname]...
    +
    +source-colname is: 
    +hive varchar column name          | 
    +trafodion varchar column name     | 
    +trafodion char column name        |
    +trafodion lob column name
    +
    +source-table is: 
    +hive table                        |
    +trafodion table 
    +```
    +
    +[#semantics]
    +=== semantics
    +
    +* `_target-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the target table in which to insert values. 
    +
    ++
    +The data type of each target column must be compatible with the data type of its corresponding source value. 
    +
    ++
    +Within the list, each target column must have the same position as its associated source value.
    +
    +** `_target-colname_` 
    ++
    +is a SQL identifier that specifies a target column name.
    +
    +* `_source-col-list-expr_`
    ++
    +Names a single column or multiple columns enclosed in parentheses in the source table from which to get values. 
    +
    +** `_source-colname_` 
    ++
    +is a SQL identifier that specifies a source column name.
    +
    +[#considerations]
    +=== Considerations
    +
    +When inserting from a source Trafodion table column into a target table, the source column subtype of the LOB column needs to match the target table column. That is, you cannot insert from an external LOB column into a regular LOB column in the target. They both need to be the same type. 
    +
    +The source for the *select clause* can be any of the following: 
    +
    +* A source hive table column that is a hive varchar column
    +* A source table column that is a Trafodion varchar, char and LOB column
    +* A source table column that is also a LOB datatype.
    +
    +[#examples]
    +=== Examples 
    +* This example inserts the first 10 rows of d_date_sk and d_date_id selected from the source hive table hive.hive.date_dim into the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2 SELECT [first 10] d_date_sk,d_date_id FROM hive.hive.date_dim;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130var into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130var;
    +```
    +
    +* This example inserts the c1 and c2 selected from the source Trafodion table t130char into the c1 and c2 of the target table t130lob2.
    ++
    +```
    +INSERT INTO t130lob2(c1,c2) SELECT c1,c2 FROM t130char;
    +```
    +
    +* This example inserts the c1, c2 and c3 selected from the source Trafodion table t130lob2 into the target table t130lob3, and then shows the result.
    ++
    +```
    +INSERT INTO t130lob3 SELECT c1,c2,c3 FROM t130lob2;
    +SELECT c1,lobtostring(c2,100),lobtostring(c3,100) FROM t130lob3;
    +```
    +
    +[#update a sql table containing lob columns]
    +== Update a SQL Table Containing LOB Columns
    +
    +The source for updating a LOB can be divided into 2 groups:
    +
    +* Update using parameters/functions
    +* Update using LOB handle
    +
    +[#update using parameters/functions]
    +=== Update Using Parameters/Functions
    +
    +The following parameters/functions can be used to update.
    +
    +* A parameter.
    +* `EMPTY_BLOB` or `EMPTY_CLOB`.
    +* An in-memory lob which is a simple string data.  
    ++
    +To insert this string, a literal needs to be provided.
    +* An on-platform file (linux/hdfs file) containing text or binary data.  
    +* A user buffer of a specified length allocated in user space.
    +* An external LOB file in HDFS.
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE table-name {set-clause-type1 | set-clause-type2} 
    +```
    +
    +```
    +set-clause-type1 is: 
    +SET set-clause[, set-clause ].. 
    +
    +set-clause is: 	
    +lob_column-name = {lob_query-expr} 
    +
    +lob_query-expr is:
    +NULL | ?                                                                                |
    +EMPTY_BLOB()                                                                            |
    +EMPTY_CLOB()                                                                            |                                                
    +STRINGTOLOB('string literal expression'[, APPEND])                                      |
    +FILETOLOB('lob source file name'[, APPEND])                                             |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value[, APPEND])      |
    +EXTERNALTOLOB('external lob source file name'[, APPEND])
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}      |  
    +{local linux file name}             |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +```
    +
    +[#semantics]
    +==== Semantics
    +
    +For more information, see Semantics in <<insert into a sql table containing lob columns,Insert into a SQL Table Containing LOB Columns>>.
    +
    +[#examples]
    +==== Examples
    +
    +* In the table tlob1where c1 is 3, updates (appends) the value of c2 to lob_update.txt.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to anoush.jpg.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('anoush.jpg');
    +```
    +
    +* In the table tlob1 where c1 is 3, updates (appends) the c2 to lob_update.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob1 SET c2=filetolob('hdfs:///lobs/lob_update.txt', append) WHERE c1 = 3;
    +```
    +
    +* In the table tlob1, updates (overwrites) the c2 to the buffer location at 1254674 with 4000-byte length.
    ++
    +```
    +UPDATE tlob1 SET c2=buffertolob(LOCATION 12546474, SIZE 4000);
    +```
    +
    +* In the table tlob130ext where c1 is 2, updates (overwrites) the c4 to lob_input_d1.txt stored in hdfs:///lobs/.
    ++
    +```
    +UPDATE tlob130ext SET c4=externaltolob('hdfs:///lobs/lob_input_d1.txt') WHERE c1=2;
    +```
    +
    +* In the table t130lob2 where c1 is 1, updates (overwrites) the c2 to xyxyxyxyxyxyx.
    ++
    +```
    +PREPARE S FROM UPDATE t130lob2 SET c2=? WHERE c1 =1;
    +EXECUTE S USING 'xyxyxyxyxyxyx';
    +```
    +
    +[# update using lob handle]
    +=== Update Using Lob Handle
    +
    +A LOB handle is specified to the update (similar to extract). 
    +
    +[#syntax]
    +==== Syntax 
    +
    +```
    +UPDATE LOB (LOB 'lob handle', lob update expression)
    +```
    +
    +```
    +lob update expression is: 
    +LOCATION lob source buffer address, LENGTH lob length value [, APPEND])      |        
    +EMPTY_BLOB()                                                                 | 
    +EMPTY_CLOB()
    +```
    +
    +For more information about examples, see http://trafodion.incubator.apache.org/docs/jdbct4ref_guide/index.html[*Trafodion JDBC Type 4 Programmer’s Reference Guide*].
    +
    +[#considerations]
    +=== Considerations
    +
    +* The keyword `'APPEND'` can be used to append to lob data that may already exist in a LOB column. If it is not specified, the data is overwritten.
    +
    +* When a LOB column is updated or overwritten, a new section is written into the LOB data file. The old section remains in HDFS but won’t have any pointer or reference to it.
    +
    +* When a LOB column is appended, a new chunk is inserted into HDFS and a new row is added in the LOB Descriptor Chunks table to keep track of this new chunk, belonging to the existing row/LOB data. 
    +
    ++
    +So a LOB column that contains data that has been appended several times can contain multiple descriptor rows in the Descriptor Chunks table to describe where each chunk belonging to this LOB begins/ends.
    +
    +* APPEND cannot be used on function `EXTERNALTOLOB`.
    +
    +* When an external LOB is updated outside of Trafodion, an update needs to be done to update the descriptor files in Trafodion. 
    ++
    +For example, if the external file changes in size, the descriptor file needs to be updated to reflect that. Since this is outside the control of Trafodion, the user needs to do this. If not, when extracting data, only partial data may be returned. 
    +
    +* If a column is declared with the `STORAGE 'External'` attribute, the `STRINGTOLOB` or `FILETOLOB` functions cannot be used to insert data into that column. 
    ++
    +Similarly, if a column is declared without the `STORAGE 'External'` attribute, the `EXTERNALTOLOB` function cannot be used to insert/update data into that column.
    ++
    +That is, once the storage for a LOB column is declared at CREATE time, the attribute is set and cannot be modified. 
    +
    +[#select column from a sql table containing lob columns]
    +== Select Column from a SQL Table Containing LOB Columns
    +
    +Selecting a row from a table will give back the lob handle for the lob column. 
    +
    +NOTE: The entry in each lob column in the SQL table only contains the LOB handle.
    +
    +Once the LOB handle has been retrieved, it can be used to retrieve the actual lob data from HDFS.
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +SELECT lob_query-expr[, lob_query-expr] FROM table-name
    +```
    +
    +```
    +lob_query_expr is: 
    +lobtostring(lob column)      |
    +lob column
    +```
    +
    +[#examples]
    +=== Examples
    +
    +* This example selects c2 from table tlob1 and returns the lob handle of c2.
    ++
    +```
    +SELECT c2 FROM tlob1;
    +C2
    +----------------------------------------------------------------------------------------------------
    +
    +LOBH00000200010423909193650389683319694857010382259683718212310961182290216021"TRAFODION"."SEABASE" 
    +
    +--- 1 row(s) selected.
    +```
    +
    +[#extract lob data from a sql table containing lob columns]
    +== Extract LOB Data from a SQL Table Containing LOB Columns 
    +
    +Extract lob data can be done in two ways:
    --- End diff --
    
    Suggest "Extracting LOB data..."


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138770574
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    --- End diff --
    
    Similar comment as with CREATE TABLE. This syntax seems to suggest that you can only have LOB columns here. But in fact you'll have other kinds of columns mixed in. Maybe something like "VALUES(column-expr[, column-expr] ...)", followed by "column-expr is: lob-query-expr | other-query-expr"


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138769765
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    --- End diff --
    
    What happens if the STORAGE clause is not present? Is there any limit on LOB size then?


---

[GitHub] incubator-trafodion pull request #1229: [TRAFODION-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r140175779
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/work_with_lob.adoc ---
    @@ -0,0 +1,738 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements. See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership. The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License. You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied. See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[#work with lob]
    += Work with LOB
    +
    +[#create a sql table with lob columns]
    +== Create a SQL Table with LOB Columns
    +
    +When creating a SQL table with LOB columns, following relevant tables and files are created as well:
    +
    +* One LOB MD table.
    +* Two dependent descriptor tables.
    +* HDFS data file (locates at /user/trafodion/lobs) for each column.
    +
    +[#syntax]
    +== Syntax
    +
    +```
    +CREATE TABLE table-name (lob-column-spec[, lob-column-spec]…)
    +```
    +
    +```
    +lob-column-spec is:
    +column {lob type}
    +
    +lob type is:
    +BLOB | CLOB [({numeric literal} [unit])] [STORAGE 'storage literal']
    +
    +unit is:
    +empty | 
    +K     | 
    +M     | 
    +G 
    +```
    +
    +[#semantics]
    +=== Semantics
    +
    +* `_storage literal_`
    +
    ++
    +Currently Trafodion only support `'EXTERNAL'` here. 
    +
    ++
    +External LOB object that are not managed by Trafodion.
    +
    +* `_empty_`
    +
    ++
    +Number of bytes specified by the numeric literal.
    +
    +* `_K_`
    +
    ++
    +Numeric literal value * 1024.
    +
    +* `_M_`
    +
    ++
    +Numeric literal value * 1024 * 1024.
    +
    +* `_G_`
    +
    ++
    +Numeric literal value * 1024 * 1024 * 1024.
    +
    +[#examples]
    +=== Examples
    +
    +* This example creates a table tlob1 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob1 (c1 INT NOT NULL, c2 BLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob2 with 3 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob2 (c1 INT NOT NULL, c2 BLOB, c3 CLOB, PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130txt_limit50 with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130txt_limit50 (c1 INT NOT NULL, c2 CLOB(50), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130bin_limit1K with 2 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130bin_limit1K (c1 INT NOT NULL, c2 BLOB(1 K), PRIMARY KEY (c1));
    +```
    +
    +* This example creates a table tlob130ext with 4 columns and primary key on the c1.
    +
    ++
    +
    +```
    +CREATE TABLE tlob130ext (c1 INT NOT NULL, c2 BLOB, c3 CLOB, c4 BLOB STORAGE 'EXTERNAL', PRIMARY KEY (c1));
    +```
    +
    +[#hdfs location of lob data]
    +=== HDFS Location of LOB Data
    +
    +When a LOB table is created, the underlying LOB data needs to be stored in HDFS.It is in the /user/trafodion/lobs by default. 
    +
    +All columns of a table that are declared as LOB types will have all their data in one file derived from the Object UID and the LOB number of that column which gets assigned during creation.
    +
    +The following is a LOB file with 2 columns you will see 2 files in HDFS:
    +
    +/user/trafodion/lobs /LOBP_03683514167332904796_0001
    +
    +/user/trafodion/lobs/LOBP_03683514167332904796_0002
    +
    +As rows are added to this column, the LOB data for each row gets appended to the corresponding column’s LOB data file. 
    +
    +[#insert into a sql table containing lob columns]
    +== Insert into a SQL Table Containing LOB Columns
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO table-name [(target-col-list)] insert-source
    +```
    +
    +```
    +target-col-list is: 
    +colname[, colname]... 
    +
    +insert-source is: 
    +VALUES(lob_query-expr[, lob_query-expr])
    +
    +lob_query-expr is: 
    +NULL | ?                                                                     |
    +EMPTY_BLOB()                                                                 |
    +EMPTY_CLOB()                                                                 |                                     
    +STRINGTOLOB('string literal expression')                                     |
    +FILETOLOB('lob source file name')                                            |
    +BUFFERTOLOB(LOCATION lob source buffer address, LENGTH lob length value)     |
    +EXTERNALTOLOB('external lob source file name')              
    +
    +lob source file name is:
    +hdfs:///{local hdfs file name}     |  
    +{local linux file name}            |
    +{file:///linux file name} 
    +
    +external lob source file name is: 
    +hdfs:///{local hdfs file name}
    +``` 
    +[#semantics]
    +=== Semantics
    +
    +* `_EMPTY_BLOB(), EMPTY_CLOB()_`
    ++
    +Returns an empty LOB handle.     
    +
    +* `_STRINGTOLOB_`
    ++
    +Converts a simple string literal into LOB format. 
    +
    +** `_string literal expression_`
    ++
    +is a series of characters enclosed in single quotes.
    +
    +* `_FILETOLOB_`
    ++
    +Converts data from a local linux/hdfs file into LOB format.
    +
    +* `_BUFFERTOLOB_`
    ++
    +Takes an address and a size of an input buffer, and converts the data pointed to by that buffer into LOB. 
    +
    +** `_lob source buffer address_`
    ++
    +The long value of the user buffer address in int64.
    +
    +** `_lob length value_`
    ++
    +The length of the user specified lob buffer in int64.
    +
    +[#considerations]
    +=== Considerations
    +
    +The source for inserting into a LOB can be any of the following:
    +
    +* A parameter.
    ++
    +An unnamed parameter can be used to prepare a statement and then during an execution, either a function or a simple string parameter can be passed in which will be converted to LOB data.
    +
    +* `EMPTY_BLOB()` or `EMPTY_CLOB()` 
    +
    +** If `EMPTY_BLOB()` or `EMPTY_CLOB()` is specified, then a dummy lob handle is created. 
    +
    +*** No data is associated with the empty LOBs yet, but these dummy LOB handles can later be used to populate with new LOB data. If the LOB had data previously associated with it, it will be erased.
    +
    +*** The dummy LOB handle will get the same datatype as the underlying column.
    ++
    +For example, if the LOB column was defined as `'EXTERNAL'` during table creation, then the LOB column gets that type. If it’s not defined, then it is considered as a regular LOB. 
    +
    +** An empty LOB is distinct from a LOB containing a string of length zero or a null LOB.
    +
    +* An in-memory LOB which is a simple string data. 
    ++
    +To insert a string literal, you need to provide `STRINGTOLOB('string literal expression')`.
    +
    +* An on-platform file (linux/hdfs file) containing binary or text data.
    ++
    +To insert an on-platform file, you need to provide `FILETOLOB('lob source file name')`.   
    +
    +* A user buffer of a specified length allocated in user space.
    ++
    +To insert a buffer, you need to provide the address and size of the buffer.
    +
    +* An external LOB.
    ++ 
    +When an external LOB is specified via `EXTERNALTOLOB('external lob source file name')`, the data associated with the external HDFS file is not transferred into the Trafodion LOB. Instead, Trafodion stores the file path/handle of the external file. 
    ++
    +For example, if you have a directory of pictures, you can specify the full hdfs path to each picture file to this function and the path will get stored in the Trafodion table. Later during retrieval, the file name will be used to go to the actual file to retrieve the data. 
    +
    +[#examples]
    +=== Examples
    +
    +* This example uses the `STRINGTOLOB` function that converts a simple string literal into LOB format before inserting.
    ++
    +```
    +INSERT INTO tlob1 VALUES(1,stringtolob('inserted row'));
    +```
    +
    +* This example uses the `FILETOLOB` function that converts data from a local file into LOB format, and stores all data into HDFS associated with that column/row.
    ++
    +```
    +INSERT INTO tlob130txt1 VALUES(1,filetolob('lob_input_a1.txt'));
    +```
    +
    +* This example takes an int64 value as an input which is an address to a buffer and a size parameter. The buffer contents are converted to LOB format and stored in HDFS.
    ++
    +```
    +INSERT INTO tlob1 VALUES (1, buffertolob(LOCATION 124647474, SIZE 2048));
    +```
    +
    +* This example uses different functions to convert strings, files, external lob into LOB data. The EXTERNALTOLOB function takes an external file. 
    ++
    +```
    +INSERT INTO tlob130ext VALUES(1, STRINGTOLOB('first lob'),                                           
    +FILETOLOB('hdfs:///lobs/lob_input_a1.txt'), 
    +EXTERNALTOLOB('hdfs:///lobs/lob_input_a1.txt'));
    +```
    +
    +* This example uses a parameter.
    ++
    +```
    +PREPARE S FROM INSERT INTO t130lob2 VALUES (1, ?);
    +EXECUTE S USING 'fgfgfhfhfhfhhfhfhfhjfkkfkffllflflfll';
    +```
    +
    +* This example uses the `EMPTY_BLOB` function to insert an empty lob and creates a dummy lob handle. 
    ++
    +```
    +INSERT INTO t130lob2 VALUES (1, empty_blob());
    +```
    +
    +[#insert into a sql table containing lob columns using select clause]
    +== Insert into a SQL Table Containing LOB Columns Using Select Clause
    +
    +[#syntax]
    +=== Syntax
    +
    +```
    +INSERT INTO target-table [(target-col-list-expr)] SELECT [source-col-list-expr] FROM source-table
    +```
    +
    +```
    +target-col-list-expr is: 
    +target-colname[, target-colname]...
    --- End diff --
    
    Thanks Dave! :)
    I've corrected.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by DaveBirdsall <gi...@git.apache.org>.
Github user DaveBirdsall commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138492998
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    +
    +[[Intended_Audience]]
    +== Intended Audience
    +This manual is intended for programmers who use LOB datatypes. 
    +
    +[[New_and_Changed_Information]]
    +== New and Changed Information
    +This is a new guide.
    +
    +[[Document_Organization]]
    +== Document Organization
    +
    +[cols="30%,70%",options="header"]
    +|===
    +|Chapter                                       | Description
    +| <<Introduction,Introduction>>                | Introduces LOB and covers following topics: +
    +&#8226; 1.1 <<what is lob,What is LOB>> +
    --- End diff --
    
    Grammar: "what is a lob,What is a LOB" is better.


---

[GitHub] incubator-trafodion pull request #1229: [Trafodion-2734] Create *Trafodion L...

Posted by liuyu000 <gi...@git.apache.org>.
Github user liuyu000 commented on a diff in the pull request:

    https://github.com/apache/incubator-trafodion/pull/1229#discussion_r138538326
  
    --- Diff: docs/lob_guide/src/asciidoc/_chapters/about.adoc ---
    @@ -0,0 +1,195 @@
    +////
    +/**
    +* @@@ START COPYRIGHT @@@
    +*
    +* Licensed to the Apache Software Foundation (ASF) under one
    +* or more contributor license agreements.  See the NOTICE file
    +* distributed with this work for additional information
    +* regarding copyright ownership.  The ASF licenses this file
    +* to you under the Apache License, Version 2.0 (the
    +* "License"); you may not use this file except in compliance
    +* with the License.  You may obtain a copy of the License at
    +*
    +*   http://www.apache.org/licenses/LICENSE-2.0
    +*
    +* Unless required by applicable law or agreed to in writing,
    +* software distributed under the License is distributed on an
    +* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
    +* KIND, either express or implied.  See the License for the
    +* specific language governing permissions and limitations
    +* under the License.
    +*
    +* @@@ END COPYRIGHT @@@
    +*/
    +////
    +
    +[[About_This_Document]]
    += About This Document
    +This guide dwscribes how to use Large Object (LOB) datatypes.
    --- End diff --
    
    Thanks Dave. 
    I've corrected. :)


---