You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@oodt.apache.org by Chris Mattmann <ma...@apache.org> on 2014/08/09 22:53:11 UTC
Review Request 24529: CAS-PGE no longer respects writers and file tags from
earlier pgeConfig.xml files
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24529/
-----------------------------------------------------------
Review request for oodt.
Bugs: OODT-667
https://issues.apache.org/jira/browse/OODT-667
Repository: oodt
Description
-------
This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE and before style configuration in which you didn't need a MIME extractor repo and could configure CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes. If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the familiar StdProductCrawler.
Diffs
-----
./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION
./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402
./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402
./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java 1616402
Diff: https://reviews.apache.org/r/24529/diff/
Testing
-------
I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this works up the point of ingestion. There is something weird going on here with InPlaceIngestion, which I'm going to take a look at, and fix, but it's pretty much done. Enjoy!
Thanks,
Chris Mattmann
Re: Review Request 24529: CAS-PGE no longer respects writers and file tags
from earlier pgeConfig.xml files
Posted by Chris Mattmann <ma...@apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24529/
-----------------------------------------------------------
(Updated Aug. 9, 2014, 10:01 p.m.)
Review request for oodt.
Changes
-------
- updated diff, now fully working and tested on DARPA XDATA.
Bugs: OODT-667
https://issues.apache.org/jira/browse/OODT-667
Repository: oodt
Description
-------
This patch restores functionality and fixes CAS-PGE in trunk for users expecting 0.3 CAS-PGE and before style configuration in which you didn't need a MIME extractor repo and could configure CAS-PGE entirely from the pge-config.xml files. This patch effectively makes CAS-PGE usable again in trunk for 0.7 and going forward and is fully forward compatible with Brian's changes. If you specify a MIME extractor repo, you get an AutoDetectCrawler - otherwise you get the familiar StdProductCrawler.
Diffs (updated)
-----
./trunk/metadata/src/main/java/org/apache/oodt/cas/metadata/filenaming/PathUtilsNamingConvention.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/PGETaskInstance.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/OutputDir.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/PgeConfigMetKeys.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/config/RegExprOutputFiles.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeMetadata.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskMetKeys.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/metadata/PgeTaskStatus.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/GenericPgeObjectFactory.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/util/XmlHelper.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/CsvConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/DynamicConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/ExternExtractorMetWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/FilenameExtractorWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/MetadataKeyReplacerTemplateWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/PcsMetFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/SciPgeConfigFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/TextConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/VelocityConfigFileWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/XslTransformWriter.java 1616402
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/metlist/MetadataListPcsMetFileWriter.java PRE-CREATION
./trunk/pge/src/main/java/org/apache/oodt/cas/pge/writers/xslt/XslTransformWriter.java PRE-CREATION
./trunk/pge/src/main/resources/examples/PgeConfigFiles/pge-config.xml 1616402
./trunk/pge/src/test/java/org/apache/oodt/cas/pge/metadata/TestPgeMetadata.java 1616402
./trunk/pge/src/test/java/org/apache/oodt/cas/pge/staging/TestFileStager.java 1616402
./trunk/pge/src/test/java/org/apache/oodt/cas/pge/writers/MockDynamicConfigFileWriter.java 1616402
Diff: https://reviews.apache.org/r/24529/diff/
Testing
-------
I've tested this on my DARPA XDATA translation ETL pipeline. Full tests are ongoing, but this works up the point of ingestion. There is something weird going on here with InPlaceIngestion, which I'm going to take a look at, and fix, but it's pretty much done. Enjoy!
Thanks,
Chris Mattmann