You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@oodt.apache.org by "Cinquini, Luca (3880)" <Lu...@jpl.nasa.gov> on 2011/03/30 20:51:41 UTC
Question about metadata extraction and file manager
Hi all,
I have a question about how to use the metadata extracted by an implementation of "FilemgrMetExtractor" when versioning a file that is ingested by the File Manager.
I have configured my File Manager to run a custom metadata extractor ("NetCDFMetExtractor") and a custom versioner ("DRSVersioner"). I am running the filemgr-client tool to ingest a netcdf file, please see the log below.
o First question: why is the versioner run twice ? It seems like the first time it is run, it has access to all the metadata that has been previously extracted by the NetCDFMetExtractor, but the second time it doesn't ?
o Second question: what is the relation between the metadata extracted by the NetCDFMetExtractor and the argument to the --metadataFile option ? Is there any way to serialize the output of the metadata extractor to a file that is then ingested by the file manager ?
thanks in advance,
Luca
=====================================================================
LMC-MAC-R46-3:bin cinquini$ ./filemgr-client --url http://localhost:9000 --operation --ingestProduct --productName ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc --productStructure Flat --productTypeName GenericFile --metadataFile file:///usr/local/cas-filemgr/bin/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc.met --refs file:///esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:08 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManagerClient <init>
INFO: Loading File Manager Configuration Properties from: [../etc/filemgr.properties]
log4j:WARN No appenders could be found for logger (org.apache.commons.httpclient.HttpClient).
log4j:WARN Please initialize the log4j system properly.
Mar 30, 2011 12:42:09 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.CoreMetExtractor] for product type: [GenericFile]
Mar 30, 2011 12:42:09 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.examples.MimeTypeExtractor] for product type: [GenericFile]
Mar 30, 2011 12:42:09 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
INFO: Running Met Extractor: [org.esgf.drs.NetCDFMetExtractor] for product type: [GenericFile]
Mar 30, 2011 12:42:09 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO:
Running NetCDF Metadata Extractor
Mar 30, 2011 12:42:09 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Parsing NetCDF file: /esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=branch_time value=0
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=parent_experiment value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=experiment_id value=decadal2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=obs_type value=satellite
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=frequency value=mon
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=obs_project value=AQUA
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=contact value=Baijun Tian, Baijun.Tian@jpl.nasa.gov
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=creation_date value=2011-03-21T20:16:36Z
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=processing_level value=L3
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=title value=AIRS model output prepared for obs4cmip5 10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=references value=http://disc.sci.gsfc.nasa.gov/AIRS/documentation http://airs.jpl.nasa.gov/documents/documents_toc/
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=experiment value=10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=institute_id value=NASA JPL
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=parent_experiment_rip value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=tracking_id value=5a257267-c9f2-4613-ada7-bd74e65754fc
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=institution value=National Aeronautics and Space Administration, Jet Propulsion Laboratory
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=forcing value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=Conventions value=CF-1.4
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=parent_experiment_id value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=table_id value=Table Amon_obs (31 January 2011) 42f63aa6a2d9531667de874a1dab7583
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=instrument value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=modeling_realm value=atmos
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=history value=converted from AIRS.2004.12.01.L3.RedStd031.v5.0.14.0.G07269214343.hdf by airs_13_2_cmip5 2011-03-21T20:16:36Z CMOR rewrote data to comply with CF standards and obs4cmip5 requirements.
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=product value=observations
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=physics_version value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=source value=AIRS (Atmospheric Infrared Sounder) retrievals
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=model_id value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=project_id value=obs4cmip5
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=realization value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=comment value=reference to technical note here
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=initialization_method value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.NetCDFMetExtractor doExtract
INFO: Extracted NetCDF metadata name=cmor_version value=2.5.3
Mar 30, 2011 12:42:13 PM org.apache.oodt.cas.filemgr.system.XmlRpcFileManager runExtractors
INFO: Running Met Extractor: [org.apache.oodt.cas.filemgr.metadata.extractors.examples.FinalFileLocationExtractor] for product type: [GenericFile]
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO:
Running DRSVersioner
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=FileLocation value=/esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=branch_time value=0
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=MimeType value=application/x-netcdf
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=ProductStructure value=Flat
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=parent_experiment value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=CAS.ProductReceivedTime value=2011-03-30T12:42:09.404-06:00
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=CAS.ProductName value=ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=experiment_id value=decadal2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=obs_type value=satellite
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=frequency value=mon
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=obs_project value=AQUA-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=contact value=Baijun Tian, Baijun.Tian@jpl.nasa.gov
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=Filename value=ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=creation_date value=2011-03-21T20:16:36Z
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=processing_level value=L3-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=title value=AIRS model output prepared for obs4cmip5 10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=references value=http://disc.sci.gsfc.nasa.gov/AIRS/documentation http://airs.jpl.nasa.gov/documents/documents_toc/
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=ProductType value=GenericFile
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=experiment value=10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=institute_id value=NASA JPL
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=parent_experiment_rip value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=tracking_id value=5a257267-c9f2-4613-ada7-bd74e65754fc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=institution value=National Aeronautics and Space Administration, Jet Propulsion Laboratory
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=Conventions value=CF-1.4
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=forcing value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=parent_experiment_id value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=obs_structure value=grid-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=table_id value=Table Amon_obs (31 January 2011) 42f63aa6a2d9531667de874a1dab7583
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=instrument value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=modeling_realm value=atmos
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=product value=observations
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=history value=converted from AIRS.2004.12.01.L3.RedStd031.v5.0.14.0.G07269214343.hdf by airs_13_2_cmip5 2011-03-21T20:16:36Z CMOR rewrote data to comply with CF standards and obs4cmip5 requirements.
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=physics_version value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=source value=AIRS (Atmospheric Infrared Sounder) retrievals
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=model_id value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=CAS.ProductId value=6b45156e-5afd-11e0-95f9-7d313f0419a5
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=project_id value=obs4cmip5
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=realization value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=initialization_method value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=comment value=reference to technical note here
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=cmor_version value=2.5.3
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioning reference=/esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=FileLocation value=/esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=branch_time value=0
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=MimeType value=application/x-netcdf
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=ProductStructure value=Flat
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=parent_experiment value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=CAS.ProductReceivedTime value=2011-03-30T12:42:09.404-06:00
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=CAS.ProductName value=ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=experiment_id value=decadal2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=obs_type value=satellite
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=frequency value=mon
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=obs_project value=AQUA-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=contact value=Baijun Tian, Baijun.Tian@jpl.nasa.gov
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=Filename value=ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=creation_date value=2011-03-21T20:16:36Z
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=processing_level value=L3-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=title value=AIRS model output prepared for obs4cmip5 10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=references value=http://disc.sci.gsfc.nasa.gov/AIRS/documentation http://airs.jpl.nasa.gov/documents/documents_toc/
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=ProductType value=GenericFile
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=experiment value=10- or 30-year run initialized in year 2000
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=institute_id value=NASA JPL
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=parent_experiment_rip value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=tracking_id value=5a257267-c9f2-4613-ada7-bd74e65754fc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=institution value=National Aeronautics and Space Administration, Jet Propulsion Laboratory
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=Conventions value=CF-1.4
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=forcing value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=parent_experiment_id value=N/A
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=obs_structure value=grid-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=table_id value=Table Amon_obs (31 January 2011) 42f63aa6a2d9531667de874a1dab7583
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=instrument value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=modeling_realm value=atmos
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=product value=observations
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=history value=converted from AIRS.2004.12.01.L3.RedStd031.v5.0.14.0.G07269214343.hdf by airs_13_2_cmip5 2011-03-21T20:16:36Z CMOR rewrote data to comply with CF standards and obs4cmip5 requirements.
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=physics_version value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=source value=AIRS (Atmospheric Infrared Sounder) retrievals
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=model_id value=AIRS
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=CAS.ProductId value=6b45156e-5afd-11e0-95f9-7d313f0419a5
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=project_id value=obs4cmip5
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=realization value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=initialization_method value=1
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=comment value=reference to technical note here
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=cmor_version value=2.5.3
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: /esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc --> file:///esg/data/cmor/obs4cmip5/observations/atmos/ta/mon/grid-new/NASA-JPL/AQUA-new/AIRS/r1i1p1/ta_AQUA-new_AIRS_L3-new_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO:
Running DRSVersioner
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=processing_level value=L3-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=obs_structure value=grid-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Using metadata key=obs_project value=AQUA-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioning reference=/esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=processing_level value=L3-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=obs_structure value=grid-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: Versioner, using metadata key=obs_project value=AQUA-new
Mar 30, 2011 12:42:13 PM org.esgf.drs.DRSVersioner createDataStoreReferences
INFO: /esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc --> file:///esg/data/cmor/obs4cmip5/observations/atmos/ta/mon/grid-new/NASA-JPL/AQUA-new/AIRS/r1i1p1/ta_AQUA-new_AIRS_L3-new_200209-201006.nc
Mar 30, 2011 12:42:13 PM org.apache.oodt.cas.filemgr.datatransfer.LocalDataTransferer moveFile
INFO: LocalDataTransfer: Moving File: file:///esg/data/cmor/obs4cmip5/observations/NASA-JPL/AIRS/decadal2000/mon/atmos/ta/r1i1p1/ta_Amon_obs_AIRS_decadal2000_r1i1p1_200209-201006.nc to file:///esg/data/cmor/obs4cmip5/observations/atmos/ta/mon/grid-new/NASA-JPL/AQUA-new/AIRS/r1i1p1/ta_AQUA-new_AIRS_L3-new_200209-201006.nc
ingestProduct: Result: 6b45156e-5afd-11e0-95f9-7d313f0419a5
LMC-MAC-R46-3:bin cinquini$
Re: Question about metadata extraction and file manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Luca,
Thanks! Yep, check out:
http://oodt.apache.org/components/maven/metadata/user/basic.html
HTH!
Cheers,
Chris
On Mar 30, 2011, at 2:06 PM, Cinquini, Luca (3880) wrote:
> Hi Chris,
> thanks for your answer - please find the policy files attached. To answer your questions:
>
> On Mar 30, 2011, at 2:55 PM, Mattmann, Chris A (388J) wrote:
>
>> Hi Luca,
>>
>> Thanks for your questions! Answers inline below:
>>
>>> I have a question about how to use the metadata extracted by an implementation of "FilemgrMetExtractor" when versioning a file that is ingested by the File Manager.
>>>
>>> I have configured my File Manager to run a custom metadata extractor ("NetCDFMetExtractor") and a custom versioner ("DRSVersioner"). I am running the filemgr-client tool to ingest a netcdf file, please see the log below.
>>>
>>> o First question: why is the versioner run twice ? It seems like the first time it is run, it has access to all the metadata that has been previously extracted by the NetCDFMetExtractor, but the second time it doesn't ?
>>
>>
>> What does your NetCDFMetExtractor do? Does it call the DRSVersioner? How are you wiring the 2 together? I see from your command line below you are attaching these to the GenericFile product type? How did you set that up? Can I see an example of your policy files? That will help to diagnose what you're seeing.
> The extractor and the versioner are not wired together via Java code, rather through the file product-types.xml, where they are both defined for the product type "GenericFile".
>>
>>>
>>> o Second question: what is the relation between the metadata extracted by the NetCDFMetExtractor and the argument to the --metadataFile option ? Is there any way to serialize the output of the metadata extractor to a file that is then ingested by the file manager ?
>>
>> The policy files will help with this. My guess is that you made the NetCDFExtractor a *server side* met extractor. You have the ability to do *client side* or *server side* extraction. From the FM client side, that's the extracted client side metadata, pre-baked in (via the --metadataFile param), or generated on the fly from a o.a.oodt.cas.metadata.extractors.CmdLineMetExtractor implementation, and piped in via the StdIngester, or via the crawler (AutoDetect, or MetExtractor).
>
> I think you probably have nailed the problem... so the metadata extractor that is defined in "product-types.xml" only affects the server side functionality ? Is there a good place where I can find how to setup the CmdLineMetExtractor + StdIngester for the client side ?
>
> thanks a lot,
> Luca
> <elements.xml><product-type-element-map.xml><product-types.xml>
>>
>> Server side met is *derived after* the original client side met is sent along during the process of ingestion.
>>
>> HTH,
>> CHris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Question about metadata extraction and file manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hey Luca,
Thanks! Yep, check out:
http://oodt.apache.org/components/maven/metadata/user/basic.html
HTH!
Cheers,
Chris
On Mar 30, 2011, at 2:06 PM, Cinquini, Luca (3880) wrote:
> Hi Chris,
> thanks for your answer - please find the policy files attached. To answer your questions:
>
> On Mar 30, 2011, at 2:55 PM, Mattmann, Chris A (388J) wrote:
>
>> Hi Luca,
>>
>> Thanks for your questions! Answers inline below:
>>
>>> I have a question about how to use the metadata extracted by an implementation of "FilemgrMetExtractor" when versioning a file that is ingested by the File Manager.
>>>
>>> I have configured my File Manager to run a custom metadata extractor ("NetCDFMetExtractor") and a custom versioner ("DRSVersioner"). I am running the filemgr-client tool to ingest a netcdf file, please see the log below.
>>>
>>> o First question: why is the versioner run twice ? It seems like the first time it is run, it has access to all the metadata that has been previously extracted by the NetCDFMetExtractor, but the second time it doesn't ?
>>
>>
>> What does your NetCDFMetExtractor do? Does it call the DRSVersioner? How are you wiring the 2 together? I see from your command line below you are attaching these to the GenericFile product type? How did you set that up? Can I see an example of your policy files? That will help to diagnose what you're seeing.
> The extractor and the versioner are not wired together via Java code, rather through the file product-types.xml, where they are both defined for the product type "GenericFile".
>>
>>>
>>> o Second question: what is the relation between the metadata extracted by the NetCDFMetExtractor and the argument to the --metadataFile option ? Is there any way to serialize the output of the metadata extractor to a file that is then ingested by the file manager ?
>>
>> The policy files will help with this. My guess is that you made the NetCDFExtractor a *server side* met extractor. You have the ability to do *client side* or *server side* extraction. From the FM client side, that's the extracted client side metadata, pre-baked in (via the --metadataFile param), or generated on the fly from a o.a.oodt.cas.metadata.extractors.CmdLineMetExtractor implementation, and piped in via the StdIngester, or via the crawler (AutoDetect, or MetExtractor).
>
> I think you probably have nailed the problem... so the metadata extractor that is defined in "product-types.xml" only affects the server side functionality ? Is there a good place where I can find how to setup the CmdLineMetExtractor + StdIngester for the client side ?
>
> thanks a lot,
> Luca
> <elements.xml><product-type-element-map.xml><product-types.xml>
>>
>> Server side met is *derived after* the original client side met is sent along during the process of ingestion.
>>
>> HTH,
>> CHris
>>
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Chris Mattmann, Ph.D.
>> Senior Computer Scientist
>> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
>> Office: 171-266B, Mailstop: 171-246
>> Email: chris.a.mattmann@nasa.gov
>> WWW: http://sunset.usc.edu/~mattmann/
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>> Adjunct Assistant Professor, Computer Science Department
>> University of Southern California, Los Angeles, CA 90089 USA
>> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>>
>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Question about metadata extraction and file manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Luca,
Thanks for your questions! Answers inline below:
> I have a question about how to use the metadata extracted by an implementation of "FilemgrMetExtractor" when versioning a file that is ingested by the File Manager.
>
> I have configured my File Manager to run a custom metadata extractor ("NetCDFMetExtractor") and a custom versioner ("DRSVersioner"). I am running the filemgr-client tool to ingest a netcdf file, please see the log below.
>
> o First question: why is the versioner run twice ? It seems like the first time it is run, it has access to all the metadata that has been previously extracted by the NetCDFMetExtractor, but the second time it doesn't ?
What does your NetCDFMetExtractor do? Does it call the DRSVersioner? How are you wiring the 2 together? I see from your command line below you are attaching these to the GenericFile product type? How did you set that up? Can I see an example of your policy files? That will help to diagnose what you're seeing.
>
> o Second question: what is the relation between the metadata extracted by the NetCDFMetExtractor and the argument to the --metadataFile option ? Is there any way to serialize the output of the metadata extractor to a file that is then ingested by the file manager ?
The policy files will help with this. My guess is that you made the NetCDFExtractor a *server side* met extractor. You have the ability to do *client side* or *server side* extraction. From the FM client side, that's the extracted client side metadata, pre-baked in (via the --metadataFile param), or generated on the fly from a o.a.oodt.cas.metadata.extractors.CmdLineMetExtractor implementation, and piped in via the StdIngester, or via the crawler (AutoDetect, or MetExtractor).
Server side met is *derived after* the original client side met is sent along during the process of ingestion.
HTH,
CHris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Re: Question about metadata extraction and file manager
Posted by "Mattmann, Chris A (388J)" <ch...@jpl.nasa.gov>.
Hi Luca,
Thanks for your questions! Answers inline below:
> I have a question about how to use the metadata extracted by an implementation of "FilemgrMetExtractor" when versioning a file that is ingested by the File Manager.
>
> I have configured my File Manager to run a custom metadata extractor ("NetCDFMetExtractor") and a custom versioner ("DRSVersioner"). I am running the filemgr-client tool to ingest a netcdf file, please see the log below.
>
> o First question: why is the versioner run twice ? It seems like the first time it is run, it has access to all the metadata that has been previously extracted by the NetCDFMetExtractor, but the second time it doesn't ?
What does your NetCDFMetExtractor do? Does it call the DRSVersioner? How are you wiring the 2 together? I see from your command line below you are attaching these to the GenericFile product type? How did you set that up? Can I see an example of your policy files? That will help to diagnose what you're seeing.
>
> o Second question: what is the relation between the metadata extracted by the NetCDFMetExtractor and the argument to the --metadataFile option ? Is there any way to serialize the output of the metadata extractor to a file that is then ingested by the file manager ?
The policy files will help with this. My guess is that you made the NetCDFExtractor a *server side* met extractor. You have the ability to do *client side* or *server side* extraction. From the FM client side, that's the extracted client side metadata, pre-baked in (via the --metadataFile param), or generated on the fly from a o.a.oodt.cas.metadata.extractors.CmdLineMetExtractor implementation, and piped in via the StdIngester, or via the crawler (AutoDetect, or MetExtractor).
Server side met is *derived after* the original client side met is sent along during the process of ingestion.
HTH,
CHris
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Chris Mattmann, Ph.D.
Senior Computer Scientist
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 171-266B, Mailstop: 171-246
Email: chris.a.mattmann@nasa.gov
WWW: http://sunset.usc.edu/~mattmann/
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Adjunct Assistant Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++