You are viewing a plain text version of this content. The canonical link for it is here.
Posted to codereview@trafodion.apache.org by selvaganesang <gi...@git.apache.org> on 2018/03/01 22:36:00 UTC

[GitHub] trafodion pull request #1461: [TRA[TRAFODION-2917] Refactor Trafodion implem...

GitHub user selvaganesang opened a pull request:

    https://github.com/apache/trafodion/pull/1461

     [TRA[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

     Removed the dependency on libhdfs in Trafodion code in some more portions of the code.
        
        Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:
        a) Prefetch the data using double buffering concept
        b) Avoid unnecessary data copy
        c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.
        
        This HDFS Scan is still being tested and hence it is turned off by default.
        
        Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/selvaganesang/trafodion libhdfs_removal

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/trafodion/pull/1461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1461
    
----
commit ac7066074611a09da33adf88673c2f023e7dda7d
Author: selvaganesang <se...@...>
Date:   2018-02-28T19:27:40Z

    [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables

commit e303b3a083154779bcde8a84e1e2abff12d365e8
Author: selvaganesang <se...@...>
Date:   2018-02-28T22:25:43Z

    [TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text formatted hive tables
    
    Removed the dependency on libhdfs in Trafodion code in some more portions of the code.
    
    Also, introduced HDFS Scan to use the java layer to read HDFS data providing the following features:
    a) Prefetch the data using double buffering concept
    b) Avoid unnecessary data copy
    c) Ensure that the data read is initiated to be read into the other buffer in java layer, while it is being processed in one buffer in the native side, without any hiccups.
    
    This HDFS Scan is still being tested and hence it is turned off by default.
    
    Also changed the code to obtain millisecond-resolution modification timestamp for HDFS files

----


---

[GitHub] trafodion pull request #1461: [TRA[TRAFODION-2917] Refactor Trafodion implem...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/trafodion/pull/1461


---