You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Venkat Ranganathan <n....@live.com> on 2015/06/24 20:35:49 UTC
Review Request 35833: FALCON-1188: Falcon support for Hive Replication
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
Review request for Falcon and Srikanth Sundarrajan.
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs
-----
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java f169917
client/src/main/java/org/apache/falcon/client/FalconClient.java 20c32e4
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
common/pom.xml 36de1f5
docs/src/site/twiki/InstallationSteps.twiki 1dd242a
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 0689899
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/#review93867
-----------------------------------------------------------
Ship it!
Ship It!
- Ajay Yadava
On July 9, 2015, 3:55 a.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35833/
> -----------------------------------------------------------
>
> (Updated July 9, 2015, 3:55 a.m.)
>
>
> Review request for Falcon and Srikanth Sundarrajan.
>
>
> Bugs: FALCON-1188
> https://issues.apache.org/jira/browse/FALCON-1188
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Falcon support for HiveDR
>
> As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
>
> It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
>
> This supports HiveDR in both secure and unsecure clusters.
>
> This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
>
> This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
>
>
> Diffs
> -----
>
> addons/hivedr/README PRE-CREATION
> addons/hivedr/pom.xml PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35833/diff/
>
>
> Testing
> -------
>
> HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
> Please see the README on component version requirements.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Ajay Yadava <aj...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/#review94891
-----------------------------------------------------------
Ship it!
Ship It!
- Ajay Yadava
On Aug. 11, 2015, 3:08 a.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35833/
> -----------------------------------------------------------
>
> (Updated Aug. 11, 2015, 3:08 a.m.)
>
>
> Review request for Falcon and Srikanth Sundarrajan.
>
>
> Bugs: FALCON-1188
> https://issues.apache.org/jira/browse/FALCON-1188
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Falcon support for HiveDR
>
> As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
>
> It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
>
> This supports HiveDR in both secure and unsecure clusters.
>
> This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
>
> This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
>
>
> Diffs
> -----
>
> Installation-steps.txt bb92a85
> addons/hivedr/README PRE-CREATION
> addons/hivedr/pom.xml PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
> addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
> addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
> addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
> addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
> addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
> addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
> client/pom.xml afa91c9
> client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789
> client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10
> client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
> client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
> client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
> client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
> client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
> docs/src/site/twiki/InstallationSteps.twiki 3dd034b
> oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
> pom.xml 34a5471
> replication/pom.xml 8c4d6b4
> src/main/assemblies/distributed-package.xml 1e48c94
> src/main/assemblies/standalone-package.xml 682d52f
> test-tools/hadoop-webapp/pom.xml cadc7f5
> webapp/pom.xml 063d42c
> webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
> webapp/src/test/resources/hdfs-replication-template.xml PRE-CREATION
> webapp/src/test/resources/hdfs-replication.properties PRE-CREATION
> webapp/src/test/resources/process.properties 1eb282c
>
> Diff: https://reviews.apache.org/r/35833/diff/
>
>
> Testing
> -------
>
> HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
> Please see the README on component version requirements.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated Aug. 10, 2015, 8:08 p.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
Patch with the recipe test case fixed (Thanks Sowmya)
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
Installation-steps.txt bb92a85
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789
client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
docs/src/site/twiki/InstallationSteps.twiki 3dd034b
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 34a5471
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/hdfs-replication-template.xml PRE-CREATION
webapp/src/test/resources/hdfs-replication.properties PRE-CREATION
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated Aug. 9, 2015, 10:46 a.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
Updated patch to use Hadoop 2.5.0. Building Oozie for Hive 1.2.0 and Hive DR instructions need more change - will follow up in a separate JIRA
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
Installation-steps.txt bb92a85
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789
client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
docs/src/site/twiki/InstallationSteps.twiki 3dd034b
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 34a5471
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated Aug. 6, 2015, 5:23 p.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
Updated patch (provided by Peeyush)
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
Installation-steps.txt bb92a85
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789
client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
common/pom.xml 36de1f5
docs/src/site/twiki/InstallationSteps.twiki 3dd034b
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 34a5471
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated Aug. 3, 2015, 11:32 p.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
Updated patch with additional pom file changes and missing files
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789
client/src/main/java/org/apache/falcon/client/FalconClient.java d507371
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
common/pom.xml 36de1f5
docs/src/site/twiki/InstallationSteps.twiki 1dd242a
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 31997e8
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/#review93982
-----------------------------------------------------------
Ship it!
- Sowmya Ramesh
On July 9, 2015, 3:55 a.m., Venkat Ranganathan wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35833/
> -----------------------------------------------------------
>
> (Updated July 9, 2015, 3:55 a.m.)
>
>
> Review request for Falcon and Srikanth Sundarrajan.
>
>
> Bugs: FALCON-1188
> https://issues.apache.org/jira/browse/FALCON-1188
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Falcon support for HiveDR
>
> As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
>
> It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
>
> This supports HiveDR in both secure and unsecure clusters.
>
> This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
>
> This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
>
>
> Diffs
> -----
>
> addons/hivedr/README PRE-CREATION
> addons/hivedr/pom.xml PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
> addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
> addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
> addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/35833/diff/
>
>
> Testing
> -------
>
> HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
> Please see the README on component version requirements.
>
>
> Thanks,
>
> Venkat Ranganathan
>
>
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated July 8, 2015, 8:55 p.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
More review fixes - cleanup locations properly handled, exceptions properly thrown etc
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan
Re: Review Request 35833: FALCON-1188: Falcon support for Hive
Replication
Posted by Venkat Ranganathan <n....@live.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------
(Updated June 26, 2015, 10:56 a.m.)
Review request for Falcon and Srikanth Sundarrajan.
Changes
-------
Updated the patch to include the followin changes.
HiveDR deletes source of root staging path after job completion
HiveDR secure workflow tmeplate is missing the DB information in the Hive2 credential URL
Bugs: FALCON-1188
https://issues.apache.org/jira/browse/FALCON-1188
Repository: falcon-git
Description
-------
Falcon support for HiveDR
As referenced in the document attached to FALCON-1188, this patch introduces Falcon support for HiveDR
It is implemented as recipe and added to Falcon addons package and can be included into the base product using the maven profile hivedr.
This supports HiveDR in both secure and unsecure clusters.
This is a large code base and we have repackage it to be in the addons so that we can quickly integrate with falcon master branch and continue further refinements/development in the master branch instead of creating a separate branch introducing merge issues
This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and Venkat Ranganathan with help from Hive folks
Diffs (updated)
-----
addons/hivedr/README PRE-CREATION
addons/hivedr/pom.xml PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/LastEvents.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION
addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION
addons/hivedr/src/main/resources/log4j.xml PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION
addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489
addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459
addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION
addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml PRE-CREATION
addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties PRE-CREATION
client/pom.xml afa91c9
client/src/main/java/org/apache/falcon/cli/FalconCLI.java f169917
client/src/main/java/org/apache/falcon/client/FalconClient.java 20c32e4
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION
client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f
client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846
client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd
client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION
common/pom.xml 36de1f5
docs/src/site/twiki/InstallationSteps.twiki 1dd242a
oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4
pom.xml 31997e8
replication/pom.xml 8c4d6b4
src/main/assemblies/distributed-package.xml 1e48c94
src/main/assemblies/standalone-package.xml 682d52f
test-tools/hadoop-webapp/pom.xml cadc7f5
webapp/pom.xml 063d42c
webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59
webapp/src/test/resources/process.properties 1eb282c
Diff: https://reviews.apache.org/r/35833/diff/
Testing
-------
HiveDR has been validated both in secure and unsecure cluster configurations. Checkstyle and RAT checks pass. There is an associated UI enhancement in Falcon UI that is provided independently.
Please see the README on component version requirements.
Thanks,
Venkat Ranganathan