You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by "Venkatesh Seetharam (JIRA)" <ji...@apache.org> on 2012/10/22 02:28:12 UTC
[jira] [Created] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Venkatesh Seetharam created SQOOP-638:
-----------------------------------------
Summary: Add an optional, simple and extensible validation framework for sqoop
Key: SQOOP-638
URL: https://issues.apache.org/jira/browse/SQOOP-638
Project: Sqoop
Issue Type: Improvement
Affects Versions: 1.4.2
Reporter: Venkatesh Seetharam
Priority: Minor
Fix For: 1.4.2
Attachments: SQOOP-638.patch
Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
There are 3 basic interfaces:
ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: SQOOP-638.patch
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: SQOOP-638.patch
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481111#comment-13481111 ]
Jarek Jarcec Cecho commented on SQOOP-638:
------------------------------------------
Hi Venkatesh,
it's quite big patch, would you mind to upload it to Apache Review Board (https://reviews.apache.org/) for easier review?
Jarcec
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508055#comment-13508055 ]
Hudson commented on SQOOP-638:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop200 #353 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/353/])
SQOOP-638: Add an optional, simple and extensible validation framework for sqoop (Revision 0b465594d24827c5a8d28e81ed3487e82937a72b)
Result = FAILURE
jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0b465594d24827c5a8d28e81ed3487e82937a72b
Files :
* src/java/org/apache/sqoop/tool/BaseSqoopTool.java
* src/java/org/apache/sqoop/validation/ValidationFailureHandler.java
* src/docs/user/validation-args.txt
* src/java/org/apache/sqoop/validation/ValidationException.java
* src/docs/user/import.txt
* src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
* src/java/org/apache/sqoop/validation/LogOnFailureHandler.java
* src/java/org/apache/sqoop/validation/Validator.java
* src/docs/user/SqoopUserGuide.txt
* src/docs/user/validation.txt
* src/java/com/cloudera/sqoop/mapreduce/JobBase.java
* src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
* src/java/org/apache/sqoop/tool/ImportTool.java
* src/java/org/apache/sqoop/validation/ValidationContext.java
* src/java/org/apache/sqoop/validation/RowCountValidator.java
* src/java/org/apache/sqoop/validation/ValidationThreshold.java
* src/docs/user/export.txt
* src/java/org/apache/sqoop/SqoopOptions.java
* src/java/org/apache/sqoop/validation/AbsoluteValidationThreshold.java
* src/docs/user/common-args.txt
* src/test/org/apache/sqoop/validation/RowCountValidatorImportTest.java
* src/java/org/apache/sqoop/tool/ExportTool.java
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.3
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508063#comment-13508063 ]
Hudson commented on SQOOP-638:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop23 #512 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/512/])
SQOOP-638: Add an optional, simple and extensible validation framework for sqoop (Revision 0b465594d24827c5a8d28e81ed3487e82937a72b)
Result = SUCCESS
jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0b465594d24827c5a8d28e81ed3487e82937a72b
Files :
* src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
* src/java/org/apache/sqoop/validation/ValidationException.java
* src/docs/user/SqoopUserGuide.txt
* src/java/com/cloudera/sqoop/mapreduce/JobBase.java
* src/docs/user/export.txt
* src/java/org/apache/sqoop/validation/ValidationFailureHandler.java
* src/java/org/apache/sqoop/tool/ExportTool.java
* src/test/org/apache/sqoop/validation/RowCountValidatorImportTest.java
* src/java/org/apache/sqoop/validation/LogOnFailureHandler.java
* src/java/org/apache/sqoop/validation/AbsoluteValidationThreshold.java
* src/java/org/apache/sqoop/validation/RowCountValidator.java
* src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
* src/docs/user/validation-args.txt
* src/java/org/apache/sqoop/SqoopOptions.java
* src/docs/user/import.txt
* src/java/org/apache/sqoop/validation/ValidationThreshold.java
* src/java/org/apache/sqoop/tool/BaseSqoopTool.java
* src/java/org/apache/sqoop/validation/ValidationContext.java
* src/java/org/apache/sqoop/validation/Validator.java
* src/java/org/apache/sqoop/tool/ImportTool.java
* src/docs/user/common-args.txt
* src/docs/user/validation.txt
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.3
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam reassigned SQOOP-638:
-----------------------------------------
Assignee: Venkatesh Seetharam
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: SQOOP-638.patch
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: SQOOP-638-r6.patch
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508059#comment-13508059 ]
Hudson commented on SQOOP-638:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop100 #347 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/347/])
SQOOP-638: Add an optional, simple and extensible validation framework for sqoop (Revision 0b465594d24827c5a8d28e81ed3487e82937a72b)
Result = SUCCESS
jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0b465594d24827c5a8d28e81ed3487e82937a72b
Files :
* src/java/org/apache/sqoop/tool/BaseSqoopTool.java
* src/docs/user/import.txt
* src/java/org/apache/sqoop/tool/ImportTool.java
* src/java/org/apache/sqoop/validation/Validator.java
* src/docs/user/validation-args.txt
* src/java/com/cloudera/sqoop/mapreduce/JobBase.java
* src/java/org/apache/sqoop/validation/RowCountValidator.java
* src/docs/user/SqoopUserGuide.txt
* src/java/org/apache/sqoop/validation/AbsoluteValidationThreshold.java
* src/test/org/apache/sqoop/validation/RowCountValidatorImportTest.java
* src/java/org/apache/sqoop/validation/ValidationFailureHandler.java
* src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
* src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
* src/docs/user/common-args.txt
* src/java/org/apache/sqoop/validation/ValidationContext.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/java/org/apache/sqoop/validation/ValidationException.java
* src/java/org/apache/sqoop/validation/LogOnFailureHandler.java
* src/java/org/apache/sqoop/tool/ExportTool.java
* src/java/org/apache/sqoop/validation/ValidationThreshold.java
* src/docs/user/export.txt
* src/docs/user/validation.txt
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.3
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Jarek Jarcec Cecho (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508058#comment-13508058 ]
Jarek Jarcec Cecho commented on SQOOP-638:
------------------------------------------
Failure in profile hadoop200 is expected due to SQOOP-731.
Jarcec
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.3
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508060#comment-13508060 ]
Hudson commented on SQOOP-638:
------------------------------
Integrated in Sqoop-ant-jdk-1.6-hadoop20 #344 (See [https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/344/])
SQOOP-638: Add an optional, simple and extensible validation framework for sqoop (Revision 0b465594d24827c5a8d28e81ed3487e82937a72b)
Result = FAILURE
jarcec : https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=0b465594d24827c5a8d28e81ed3487e82937a72b
Files :
* src/java/org/apache/sqoop/mapreduce/ExportJobBase.java
* src/java/org/apache/sqoop/tool/ImportTool.java
* src/java/org/apache/sqoop/validation/LogOnFailureHandler.java
* src/java/org/apache/sqoop/validation/ValidationContext.java
* src/docs/user/common-args.txt
* src/java/org/apache/sqoop/tool/BaseSqoopTool.java
* src/java/org/apache/sqoop/validation/Validator.java
* src/docs/user/validation.txt
* src/docs/user/import.txt
* src/docs/user/export.txt
* src/java/org/apache/sqoop/validation/ValidationFailureHandler.java
* src/java/org/apache/sqoop/mapreduce/ImportJobBase.java
* src/docs/user/SqoopUserGuide.txt
* src/java/org/apache/sqoop/tool/ExportTool.java
* src/docs/user/validation-args.txt
* src/java/org/apache/sqoop/validation/RowCountValidator.java
* src/java/com/cloudera/sqoop/mapreduce/JobBase.java
* src/java/org/apache/sqoop/SqoopOptions.java
* src/java/org/apache/sqoop/validation/ValidationException.java
* src/java/org/apache/sqoop/validation/AbsoluteValidationThreshold.java
* src/java/org/apache/sqoop/validation/ValidationThreshold.java
* src/test/org/apache/sqoop/validation/RowCountValidatorImportTest.java
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.3
>
> Attachments: SQOOP-638.patch, SQOOP-638-r6.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: (was: SQOOP-638.patch)
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: (was: SQOOP-638.patch)
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: SQOOP-638.patch
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (SQOOP-638) Add an optional, simple and extensible
validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Venkatesh Seetharam updated SQOOP-638:
--------------------------------------
Attachment: (was: SQOOP-638.patch)
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (SQOOP-638) Add an optional, simple and
extensible validation framework for sqoop
Posted by "Venkatesh Seetharam (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SQOOP-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481183#comment-13481183 ]
Venkatesh Seetharam commented on SQOOP-638:
-------------------------------------------
Sorry that I forgot to put it on RB. Please find the link: https://reviews.apache.org/r/7693/
> Add an optional, simple and extensible validation framework for sqoop
> ---------------------------------------------------------------------
>
> Key: SQOOP-638
> URL: https://issues.apache.org/jira/browse/SQOOP-638
> Project: Sqoop
> Issue Type: Improvement
> Affects Versions: 1.4.2
> Reporter: Venkatesh Seetharam
> Assignee: Venkatesh Seetharam
> Priority: Minor
> Fix For: 1.4.2
>
> Attachments: SQOOP-638.patch
>
>
> Attempt to add an extensible validation framework to Sqoop. Adds an optional CLI option: --validate
> There are 3 basic interfaces:
> ValidationThreshold - Determines if the error margin between the source and target are acceptable: Absolute, Percentage Tolerant, etc.
> Default implementation is AbsoluteValidationThreshold which ensures the row counts from source and targets are the same.
> ValidationFailureHandler - Responsible for handling failures: log an error/warning, abort, etc. Default implementation logs a warning message to the configured logger.
> Validator - Drives the validation logic by delegating the decision to ValidationThreshold and delegating failure handling to ValidationFailureHandler. The default implementation comes with a RowCountValidator which validates the row counts from source and the target.
> You could extend these interfaces for more specific implementations and override 'em in sqoop configuration that is picked up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira