[jira] [Commented] (TRAFODION-1473) Adding external buffer and file input and extract support for LOB datatypes

["Sandhya Sundaresan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/TRAFODION-1473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14903186#comment-14903186 ] 

Sandhya Sundaresan commented on TRAFODION-1473:
-----------------------------------------------

Added support for the following :
•	Support filetolob  function to input file data into a lob .
•	Support for Extract syntax to extract  length of a given lob.
•	Support for Extract syntax to extract data given a lob handle. This includes support for lobtoFile function to extract the lobdata into a file.
Various syntaxes :
Extract lobtofile (lob <lob handle> , <filename>) ; if file exists error, if file does not exist create.
Extract lobtofile (lob <lob handle> , <filename>, truncate ) ; if file exists truncater, if file does not exist create.
Extract lobtofile (lob <lob handle> , <filename>, create, append ) ; if file exists append, if file does not exist error
Extract lobtofile (lob <lob handle> , <filename>, create, append ) ; if file exists append, if file does not exist create.

•	Remove the 2G limitation for lob data files.
•	Support for extract from a lob column to an external file(on platform linux or hdfs file) in chunks .
•	Support for input from an external file (on platform linux or hdfs file) in chunks into lob column.
•	Changed semantics of lob  extract to file to match  hdfs’s APIs (copyToLocalFile)
•	Lob Max size is now a CQD and configurable
•	Lob Max Chunk Size limits how much data is used to allocate and write in chunks. Earlier the chunk size was sixed and now it is configurable with a CQD.
•	Earlier the lob extract size was limited to LOB_OUTPUT_SIZE which was 32K
•	Now it’s limited by LOB_MAX_CHUNK_MEM_SIZE but can also be specified by the caller in syntax.  (there is a check to ensure it is less than LOB_MX_CHUNK_MEM_SIZE).

•	Added Regression test to:

1) insert into lob column using :
a)	local text file.
b)	hdfs text file
c)	local binary file (jpg file)
d)	hdfs binary format file (jpg file)

2) Regression test to extract from lob column  into :
a)	Local text file
b)	Local hdfs text file
c)	Local binary file (jpg file)
d)	Hdfs binary file (jpg file)




> Adding external buffer and file input and extract support  for  LOB datatypes
> -----------------------------------------------------------------------------
>
>                 Key: TRAFODION-1473
>                 URL: https://issues.apache.org/jira/browse/TRAFODION-1473
>             Project: Apache Trafodion
>          Issue Type: Task
>          Components: sql-cmp, sql-exe
>         Environment: Trafodion
>            Reporter: Sandhya Sundaresan
>            Assignee: Sandhya Sundaresan
>              Labels: features
>             Fix For: 2.0-incubating
>
>
> LOB datatype is disabled in Trafodion  until the following support is fully in.
> Here are the list of tasks and items to support :
> 1. Support input of linux files (on platform)  into LOB columns.
> 2. Support extract of LOB data into linux files on platform.
> 3. Support input of hdfs files  into LOB columns.
> 4. Support extract of LOB data into hdfs files.
> 5. Support chunking to input a very large external file into LOB columns.
> 6. Use CQD concept to limit LOB max size and limit chunk size for extract /input.
> TBD 
> 7. Support for  input/extract of external files off platform into LOB columns.
> 8. Support in connectivity and driver for real LOB data type.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)