You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@falcon.apache.org by Shwetha GS <ss...@hortonworks.com> on 2015/02/24 08:40:11 UTC
Review Request 31343: FALCON-703 Monitoring plugin that registers
catalog partition
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31343/
-----------------------------------------------------------
Review request for Falcon.
Bugs: FALCON-703
https://issues.apache.org/jira/browse/FALCON-703
Repository: falcon-git
Description
-------
Added JMS listener that registers partitions for filesystem based feeds. Adding the first version
Pending:
1. verify existing UTs
2. Add more tests
Diffs
-----
common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93
common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION
common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46
common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b
common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95
common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java fe93048
common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb
common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java e04f046
common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java bbd73c7
common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a
common/src/main/resources/startup.properties 433c2a8
common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556
common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc
common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161
hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java 2167375
oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb
oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 7a87919
oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java ce292bd
prism/src/main/java/org/apache/falcon/Main.java de9f657
prism/src/main/java/org/apache/falcon/listener/ContextStartupListener.java 5e879b1
prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java 788df58
prism/src/main/java/org/apache/falcon/util/SecureEmbeddedServer.java 8e9ef52
test-tools/hadoop-webapp/src/main/java/org/apache/falcon/listener/HadoopStartupListener.java 7bee592
test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de
test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b
webapp/pom.xml 556b171
webapp/src/conf/oozie/conf/hadoop-conf/core-site.xml a0f1e07
webapp/src/conf/oozie/conf/hadoop-conf/mapred-site.xml 96574b7
webapp/src/conf/oozie/conf/oozie-site.xml ded4873
webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION
webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9
webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9
webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a
webapp/src/test/resources/cluster-template.xml 16b7c8c
webapp/src/test/resources/feed-template1.xml 456f7ce
webapp/src/test/resources/feed-template2.xml d4901fa
Diff: https://reviews.apache.org/r/31343/diff/
Testing
-------
Manual end to end test, added one end to end success case
Thanks,
Shwetha GS
Re: Review Request 31343: FALCON-703 Monitoring plugin that registers
catalog partition
Posted by Shwetha GS <ss...@hortonworks.com>.
> On March 5, 2015, 8:56 a.m., Suhas Vasu wrote:
> > prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java, line 22
> > <https://reviews.apache.org/r/31343/diff/1/?file=873512#file873512line22>
> >
> > out of curiousity, why are we changing this ?
Not required now. Was trying to upgrade jetty version as something wasn't working in IT
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31343/#review75306
-----------------------------------------------------------
On Feb. 24, 2015, 7:40 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31343/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 7:40 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-703
> https://issues.apache.org/jira/browse/FALCON-703
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Added JMS listener that registers partitions for filesystem based feeds. Adding the first version
>
> Pending:
> 1. verify existing UTs
> 2. Add more tests
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION
> common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46
> common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95
> common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java fe93048
> common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb
> common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java e04f046
> common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java bbd73c7
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a
> common/src/main/resources/startup.properties 433c2a8
> common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556
> common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc
> common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161
> hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java 2167375
> oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 7a87919
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java ce292bd
> prism/src/main/java/org/apache/falcon/Main.java de9f657
> prism/src/main/java/org/apache/falcon/listener/ContextStartupListener.java 5e879b1
> prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java 788df58
> prism/src/main/java/org/apache/falcon/util/SecureEmbeddedServer.java 8e9ef52
> test-tools/hadoop-webapp/src/main/java/org/apache/falcon/listener/HadoopStartupListener.java 7bee592
> test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de
> test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b
> webapp/pom.xml 556b171
> webapp/src/conf/oozie/conf/hadoop-conf/core-site.xml a0f1e07
> webapp/src/conf/oozie/conf/hadoop-conf/mapred-site.xml 96574b7
> webapp/src/conf/oozie/conf/oozie-site.xml ded4873
> webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9
> webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9
> webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a
> webapp/src/test/resources/cluster-template.xml 16b7c8c
> webapp/src/test/resources/feed-template1.xml 456f7ce
> webapp/src/test/resources/feed-template2.xml d4901fa
>
> Diff: https://reviews.apache.org/r/31343/diff/
>
>
> Testing
> -------
>
> Manual end to end test, added one end to end success case
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 31343: FALCON-703 Monitoring plugin that registers
catalog partition
Posted by Pallavi Rao <pa...@inmobi.com>.
> On March 5, 2015, 8:56 a.m., Suhas Vasu wrote:
> > common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java, line 98
> > <https://reviews.apache.org/r/31343/diff/1/?file=873493#file873493line98>
> >
> > may be i am wrong, will there be no scenario where feednames & feedpaths may not be a 1-1 mapping ?
Suhas,
I looked at ProcessExecutionCoordinatorBuilder where this variable is populated. The order of feednames and feedpaths match. As an additional shield, we can check to make sure the basePath of the output instance path matches the one in the feed definition. Makes sense?
- Pallavi
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31343/#review75306
-----------------------------------------------------------
On Feb. 24, 2015, 7:40 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31343/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 7:40 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-703
> https://issues.apache.org/jira/browse/FALCON-703
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Added JMS listener that registers partitions for filesystem based feeds. Adding the first version
>
> Pending:
> 1. verify existing UTs
> 2. Add more tests
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION
> common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46
> common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95
> common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java fe93048
> common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb
> common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java e04f046
> common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java bbd73c7
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a
> common/src/main/resources/startup.properties 433c2a8
> common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556
> common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc
> common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161
> hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java 2167375
> oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 7a87919
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java ce292bd
> prism/src/main/java/org/apache/falcon/Main.java de9f657
> prism/src/main/java/org/apache/falcon/listener/ContextStartupListener.java 5e879b1
> prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java 788df58
> prism/src/main/java/org/apache/falcon/util/SecureEmbeddedServer.java 8e9ef52
> test-tools/hadoop-webapp/src/main/java/org/apache/falcon/listener/HadoopStartupListener.java 7bee592
> test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de
> test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b
> webapp/pom.xml 556b171
> webapp/src/conf/oozie/conf/hadoop-conf/core-site.xml a0f1e07
> webapp/src/conf/oozie/conf/hadoop-conf/mapred-site.xml 96574b7
> webapp/src/conf/oozie/conf/oozie-site.xml ded4873
> webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9
> webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9
> webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a
> webapp/src/test/resources/cluster-template.xml 16b7c8c
> webapp/src/test/resources/feed-template1.xml 456f7ce
> webapp/src/test/resources/feed-template2.xml d4901fa
>
> Diff: https://reviews.apache.org/r/31343/diff/
>
>
> Testing
> -------
>
> Manual end to end test, added one end to end success case
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 31343: FALCON-703 Monitoring plugin that registers
catalog partition
Posted by Shwetha GS <ss...@hortonworks.com>.
> On March 5, 2015, 8:56 a.m., Suhas Vasu wrote:
> > common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java, line 98
> > <https://reviews.apache.org/r/31343/diff/1/?file=873493#file873493line98>
> >
> > may be i am wrong, will there be no scenario where feednames & feedpaths may not be a 1-1 mapping ?
>
> Pallavi Rao wrote:
> Suhas,
> I looked at ProcessExecutionCoordinatorBuilder where this variable is populated. The order of feednames and feedpaths match. As an additional shield, we can check to make sure the basePath of the output instance path matches the one in the feed definition. Makes sense?
I think feed retention messages don't honour this
- Shwetha
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31343/#review75306
-----------------------------------------------------------
On Feb. 24, 2015, 7:40 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31343/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 7:40 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-703
> https://issues.apache.org/jira/browse/FALCON-703
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Added JMS listener that registers partitions for filesystem based feeds. Adding the first version
>
> Pending:
> 1. verify existing UTs
> 2. Add more tests
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION
> common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46
> common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95
> common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java fe93048
> common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb
> common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java e04f046
> common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java bbd73c7
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a
> common/src/main/resources/startup.properties 433c2a8
> common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556
> common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc
> common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161
> hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java 2167375
> oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 7a87919
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java ce292bd
> prism/src/main/java/org/apache/falcon/Main.java de9f657
> prism/src/main/java/org/apache/falcon/listener/ContextStartupListener.java 5e879b1
> prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java 788df58
> prism/src/main/java/org/apache/falcon/util/SecureEmbeddedServer.java 8e9ef52
> test-tools/hadoop-webapp/src/main/java/org/apache/falcon/listener/HadoopStartupListener.java 7bee592
> test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de
> test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b
> webapp/pom.xml 556b171
> webapp/src/conf/oozie/conf/hadoop-conf/core-site.xml a0f1e07
> webapp/src/conf/oozie/conf/hadoop-conf/mapred-site.xml 96574b7
> webapp/src/conf/oozie/conf/oozie-site.xml ded4873
> webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9
> webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9
> webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a
> webapp/src/test/resources/cluster-template.xml 16b7c8c
> webapp/src/test/resources/feed-template1.xml 456f7ce
> webapp/src/test/resources/feed-template2.xml d4901fa
>
> Diff: https://reviews.apache.org/r/31343/diff/
>
>
> Testing
> -------
>
> Manual end to end test, added one end to end success case
>
>
> Thanks,
>
> Shwetha GS
>
>
Re: Review Request 31343: FALCON-703 Monitoring plugin that registers
catalog partition
Posted by Suhas Vasu <su...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31343/#review75306
-----------------------------------------------------------
common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java
<https://reviews.apache.org/r/31343/#comment122291>
Java doc missing ?
common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java
<https://reviews.apache.org/r/31343/#comment122292>
Java doc missing ?
common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java
<https://reviews.apache.org/r/31343/#comment122297>
Does it make sense to change it to StringUtils.isEmpty() ?
common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java
<https://reviews.apache.org/r/31343/#comment122298>
may be i am wrong, will there be no scenario where feednames & feedpaths may not be a 1-1 mapping ?
prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java
<https://reviews.apache.org/r/31343/#comment122300>
out of curiousity, why are we changing this ?
- Suhas Vasu
On Feb. 24, 2015, 7:40 a.m., Shwetha GS wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31343/
> -----------------------------------------------------------
>
> (Updated Feb. 24, 2015, 7:40 a.m.)
>
>
> Review request for Falcon.
>
>
> Bugs: FALCON-703
> https://issues.apache.org/jira/browse/FALCON-703
>
>
> Repository: falcon-git
>
>
> Description
> -------
>
> Added JMS listener that registers partitions for filesystem based feeds. Adding the first version
>
> Pending:
> 1. verify existing UTs
> 2. Add more tests
>
>
> Diffs
> -----
>
> common/src/main/java/org/apache/falcon/catalog/AbstractCatalogService.java 9abdc93
> common/src/main/java/org/apache/falcon/catalog/CatalogPartitionHandler.java PRE-CREATION
> common/src/main/java/org/apache/falcon/catalog/HiveCatalogService.java 25a4a46
> common/src/main/java/org/apache/falcon/entity/CatalogStorage.java 59f558b
> common/src/main/java/org/apache/falcon/entity/FeedHelper.java ca31f95
> common/src/main/java/org/apache/falcon/entity/FileSystemStorage.java fe93048
> common/src/main/java/org/apache/falcon/entity/common/FeedDataPath.java 6ededbb
> common/src/main/java/org/apache/falcon/expression/ExpressionHelper.java e04f046
> common/src/main/java/org/apache/falcon/util/FalconRadixUtils.java bbd73c7
> common/src/main/java/org/apache/falcon/workflow/WorkflowExecutionContext.java 8d69b9a
> common/src/main/resources/startup.properties 433c2a8
> common/src/test/java/org/apache/falcon/entity/FeedDataPathTest.java c405556
> common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java f6994fc
> common/src/test/java/org/apache/falcon/entity/FileSystemStorageTest.java 1667161
> hadoop-dependencies/src/versioned-src/v2/java/org/apache/hadoop/mapred/ClassicClientProtocolProvider.java 2167375
> oozie/src/main/java/org/apache/falcon/logging/LogProvider.java 2e5dffb
> oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java 7a87919
> oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java ce292bd
> prism/src/main/java/org/apache/falcon/Main.java de9f657
> prism/src/main/java/org/apache/falcon/listener/ContextStartupListener.java 5e879b1
> prism/src/main/java/org/apache/falcon/util/EmbeddedServer.java 788df58
> prism/src/main/java/org/apache/falcon/util/SecureEmbeddedServer.java 8e9ef52
> test-tools/hadoop-webapp/src/main/java/org/apache/falcon/listener/HadoopStartupListener.java 7bee592
> test-tools/hadoop-webapp/src/main/resources/mapred-site.xml cf297de
> test-tools/hadoop-webapp/src/main/resources/yarn-site.xml 658752b
> webapp/pom.xml 556b171
> webapp/src/conf/oozie/conf/hadoop-conf/core-site.xml a0f1e07
> webapp/src/conf/oozie/conf/hadoop-conf/mapred-site.xml 96574b7
> webapp/src/conf/oozie/conf/oozie-site.xml ded4873
> webapp/src/test/java/org/apache/falcon/catalog/CatalogPartitionHandlerIT.java PRE-CREATION
> webapp/src/test/java/org/apache/falcon/catalog/HiveCatalogServiceIT.java 71616e9
> webapp/src/test/java/org/apache/falcon/util/HiveTestUtils.java 19274b9
> webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java e67fe2a
> webapp/src/test/resources/cluster-template.xml 16b7c8c
> webapp/src/test/resources/feed-template1.xml 456f7ce
> webapp/src/test/resources/feed-template2.xml d4901fa
>
> Diff: https://reviews.apache.org/r/31343/diff/
>
>
> Testing
> -------
>
> Manual end to end test, added one end to end success case
>
>
> Thanks,
>
> Shwetha GS
>
>