[jira] [Comment Edited] (SQOOP-1168) Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )

["Veena Basavaraj (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SQOOP-1168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14277287#comment-14277287 ] 

Veena Basavaraj edited comment on SQOOP-1168 at 1/14/15 5:41 PM:
-----------------------------------------------------------------

For Question 2:

Well  INCREMENTAL_NEW_ROWS_ONLY ( even if we change it to something else) , does it make it any less confusing to the users of sqoop2, I am not convinced it does add much value.
Sqoop2 is not sqoop1 and if are trying to retrofit a terminology from sqoop1 to sqoop2, there are other serious implications, what does INCREMENTAL_NEW_ROWS_ONLY mean for kafka or a kinesis connector.

As for using Enums, it is a connector only knowledge, and sqoop need not be aware of it. So I do not follow the rest of conversation on upgrade nor relevant values. The proposal clearly says ( and with examples) that a user is aware of what they choose and connector is aware of what it does. Sqoop is and SHOULD be opaque to this. A connector controls what its enum values are, I do not even think we need a common language as it is suggested clearly, again to reiterate what a connector exposes as configs and how it changes upgrades them is upto it now.  

But if we are go with a Enum for DeltaMerge ( Again I do not think Incremental makes any sense to keep using as a term in sqoop2 since it does not apply to the Sqoop2 generic design and nor does this term apply to all data sources), We have format and CompressionForm Enum today for HDFS, so if a connector uses a ENUM for delta write/read it will be very much along that line

{code}
  public static String DEFAULT_NULL_VALUE = "NULL";

  @Input(size = 255) public Boolean overrideNullValue;

  @Input(size = 255) public String nullValue;

  @Input public ToFormat outputFormat;

  @Input public ToCompression compression;

  @Input(size = 255) public String customCompression;

  @Input(size = 255, validators = { @Validator(NotEmpty.class)}) public String outputDirectory;

{code}

{code}
/
package org.apache.sqoop.connector.hdfs.configuration;

/**
 * Supported compressions
 */
public enum ToCompression {
  NONE,
  DEFAULT,
  DEFLATE,
  GZIP,
  BZIP2,
  LZO,
  LZ4,
  SNAPPY,
  CUSTOM,
}
{code}




was (Author: vybs):
For Question 2:

Well  INCREMENTAL_NEW_ROWS_ONLY ( even if we change it to something else) , does it make it any less confusing to the users of sqoop2, I am not convinced it does add much value.
Sqoop2 is not sqoop1 and if are trying to retrofit a terminology from sqoop1 to sqoop2, there are other serious implications, what does INCREMENTAL_NEW_ROWS_ONLY mean for kafka or a kinesis connector.

As for using Enums, it is a connector only knowledge, and sqoop need not be aware of it. So I do not follow the rest of conversation on upgrade nor relevant values. The proposal clearly says ( and with examples) that a user is aware of what they choose and connector is aware of what it does. Sqoop is and SHOULD be opaque to this. A connector controls what its enum values are, I do not even think we need a common language as it is suggested clearly, again to reiterate what a connector exposes as configs and how it changes upgrades them is upto it now.  

But if we are go with a Enum for DeltaMerge ( Again I do not think Incremental makes any sense to keep using as a term in sqoop2 since it does not apply to the Sqoop2 generic design and nor does this term apply to all data sources), We have format and CompressionForm Enum today for HDFS, so if a connector uses a ENUM for delta write/read it will be very much along that line



> Sqoop2: Delta Fetch/ Merge ( formerly called Incremental Import )
> -----------------------------------------------------------------
>
>                 Key: SQOOP-1168
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1168
>             Project: Sqoop
>          Issue Type: Bug
>            Reporter: Hari Shreedharan
>            Assignee: Veena Basavaraj
>             Fix For: 1.99.6
>
>
> The formal design wiki is here 
> https://cwiki.apache.org/confluence/display/SQOOP/Delta+Fetch+and+Merge+Design



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)