You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cmda.apache.org by "Lei Pan (JIRA)" <ji...@apache.org> on 2016/03/21 18:24:25 UTC
[jira] [Updated] (CMDA-9) Modification on entry.py to enable
climate services rerun on CMU provenance
[ https://issues.apache.org/jira/browse/CMDA-9?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lei Pan updated CMDA-9:
-----------------------
Description:
Here is a modification request from CMU team.
The reason why we could get plot and re-run with the same configuration from CMU side but not on JPL side is that we actually modified entry.py file on Einstein for testing purpose.
In order to make re-run pages work as well as enabling automatic image and .nc files downloading once for all, I may need to request you to make the following changes on entry.py.
1.Modify
===============================================================
post_json_wei = {'dataUrl': dataUrl, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
'parameters': parameters_json, 'url': url, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
===============================================================
TO
===============================================================
post_json_wei = {'plotUrl':plotUrl, 'dataUrl':dataUrl, 'urlLink':urlLink, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
'parameters': parameters_json, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
===============================================================
Where value of urlLink should equal to, for example
===============================================================
"?model1=GFDL_ESM2G&var1=pr&lon1=0&lon2=360&lat1=-90&lat2=90&startT=200401&endT=200412&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&Image=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg&data_url=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc&purpose="
===============================================================
We will store that string in our database and when a user need to re-run one service through execution log, we will utilize the string since Benyang has already enabled such function in HTML pages.
And dataUrl should be, for example, "http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc", which is the location of OutputFile.
And plotUrl should be, for example, ""http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg", which is the location of image generated.
2.REMOVE
===============================================================
post_json_addPicture = {'url':url}
post_json_addPicture = json.dumps(post_json_addPicture)
post_json_addOutputfile = {'url':dataUrl}
post_json_addOutputfile = json.dumps(post_json_addOutputfile)
print 'post_json_addPicture: ', post_json_addPicture
print 'post_json_addOutputfile: ', post_json_addOutputfile
===============================================================
And
===============================================================
print requests.post(BASE_POST_URL_ADD_PICTURE, data=post_json_addPicture, headers=HEADERS).text
print requests.post(BASE_POST_URL_ADD_OUTPUTFILE, data=post_json_addOutputfile, headers=HEADERS).text
===============================================================
Because we merge 3 APIs into one API.
============= from Lei to Qihao, Sunday, March 20, 2016 at 11:53 PM ================
Hi Qihao:
I have put in the new way to call your provenance service
in out 2D Var Map service, and issued a request. The urlLink
and the returning status are as follows. Could you please
confirm that your service has indeed received the image,
the .nc file, as well as all the input parameter? The execution
purpose I put in is "test the new call to the CMU provenance service,
by Lei, 3/20/2016."
Thanks,
-Lei
urlLink: model1=GFDL_ESM2G&var1=pr&lon1=0&lon2=360&lat1=-90&lat2=90&startT=200401&endT=200412&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&image=http://cmda-test.jpl.nasa.gov:8090/static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.jpeg&data_url=http://cmda-test.jpl.nasa.gov:8090/static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.nc
104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /svc/twoDimMap?model=GFDL_ESM2G&var=pr&start_time=200401&end_time=200412&lon1=0&lon2=360&lat1=-90&lat2=90&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&purpose=test%20the%20new%20call%20to%20the%20CMU%20provenance%20service,%20by%20Lei,%203/20/2016. HTTP/1.1" 200 -
source: GFDL/ESM2G
var: pr
/mnt/data_2015/cmip5/gfdl/esm2g/./pr_*.nc
retDateList: [199601, 200512]
Wei's URL called here
104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /svc/time_bounds?serviceType=1&source=GFDL/ESM2G&var=pr HTTP/1.1" 200 -
104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.jpeg HTTP/1.1" 200 -
was:
Here is a modification request from CMU team.
The reason why we could get plot and re-run with the same configuration from CMU side but not on JPL side is that we actually modified entry.py file on Einstein for testing purpose.
In order to make re-run pages work as well as enabling automatic image and .nc files downloading once for all, I may need to request you to make the following changes on entry.py.
1.Modify
===============================================================
post_json_wei = {'dataUrl': dataUrl, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
'parameters': parameters_json, 'url': url, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
===============================================================
TO
===============================================================
post_json_wei = {'plotUrl':plotUrl, 'dataUrl':dataUrl, 'urlLink':urlLink, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
'parameters': parameters_json, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
===============================================================
Where value of urlLink should equal to, for example
===============================================================
"?model1=GFDL_ESM2G&var1=pr&lon1=0&lon2=360&lat1=-90&lat2=90&startT=200401&endT=200412&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&Image=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg&data_url=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc&purpose="
===============================================================
We will store that string in our database and when a user need to re-run one service through execution log, we will utilize the string since Benyang has already enabled such function in HTML pages.
And dataUrl should be, for example, "http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc", which is the location of OutputFile.
And plotUrl should be, for example, ""http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg", which is the location of image generated.
2.REMOVE
===============================================================
post_json_addPicture = {'url':url}
post_json_addPicture = json.dumps(post_json_addPicture)
post_json_addOutputfile = {'url':dataUrl}
post_json_addOutputfile = json.dumps(post_json_addOutputfile)
print 'post_json_addPicture: ', post_json_addPicture
print 'post_json_addOutputfile: ', post_json_addOutputfile
===============================================================
And
===============================================================
print requests.post(BASE_POST_URL_ADD_PICTURE, data=post_json_addPicture, headers=HEADERS).text
print requests.post(BASE_POST_URL_ADD_OUTPUTFILE, data=post_json_addOutputfile, headers=HEADERS).text
===============================================================
Because we merge 3 APIs into one API.
> Modification on entry.py to enable climate services rerun on CMU provenance
> ---------------------------------------------------------------------------
>
> Key: CMDA-9
> URL: https://issues.apache.org/jira/browse/CMDA-9
> Project: Apache Climate Model Diagnostic Analyzer
> Issue Type: Task
> Environment: entry.py
> Reporter: Qihao Bao
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Here is a modification request from CMU team.
> The reason why we could get plot and re-run with the same configuration from CMU side but not on JPL side is that we actually modified entry.py file on Einstein for testing purpose.
> In order to make re-run pages work as well as enabling automatic image and .nc files downloading once for all, I may need to request you to make the following changes on entry.py.
> 1.Modify
> ===============================================================
> post_json_wei = {'dataUrl': dataUrl, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
> 'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
> 'parameters': parameters_json, 'url': url, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
> ===============================================================
> TO
> ===============================================================
> post_json_wei = {'plotUrl':plotUrl, 'dataUrl':dataUrl, 'urlLink':urlLink, 'userId': long(userId), 'serviceId':long(serviceId), 'purpose':purpose,
> 'executionStartTime':long(executionStartTime)*1000, 'executionEndTime':long(executionEndTime)*1000,
> 'parameters': parameters_json, 'datasetStudyStartTime': startT, 'datasetStudyEndTime': endT}
> ===============================================================
> Where value of urlLink should equal to, for example
> ===============================================================
> "?model1=GFDL_ESM2G&var1=pr&lon1=0&lon2=360&lat1=-90&lat2=90&startT=200401&endT=200412&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&Image=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg&data_url=http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc&purpose="
> ===============================================================
> We will store that string in our database and when a user need to re-run one service through execution log, we will utilize the string since Benyang has already enabled such function in HTML pages.
> And dataUrl should be, for example, "http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.nc", which is the location of OutputFile.
> And plotUrl should be, for example, ""http://cmacws4.jpl.nasa.gov:8090/static/twoDimZonalMean/3035de17274428938127423b6029c754/gfdl_esm2g_pr_200401_200412_Annual.jpeg", which is the location of image generated.
> 2.REMOVE
> ===============================================================
> post_json_addPicture = {'url':url}
> post_json_addPicture = json.dumps(post_json_addPicture)
> post_json_addOutputfile = {'url':dataUrl}
> post_json_addOutputfile = json.dumps(post_json_addOutputfile)
> print 'post_json_addPicture: ', post_json_addPicture
> print 'post_json_addOutputfile: ', post_json_addOutputfile
> ===============================================================
> And
> ===============================================================
> print requests.post(BASE_POST_URL_ADD_PICTURE, data=post_json_addPicture, headers=HEADERS).text
> print requests.post(BASE_POST_URL_ADD_OUTPUTFILE, data=post_json_addOutputfile, headers=HEADERS).text
> ===============================================================
> Because we merge 3 APIs into one API.
> ============= from Lei to Qihao, Sunday, March 20, 2016 at 11:53 PM ================
> Hi Qihao:
> I have put in the new way to call your provenance service
> in out 2D Var Map service, and issued a request. The urlLink
> and the returning status are as follows. Could you please
> confirm that your service has indeed received the image,
> the .nc file, as well as all the input parameter? The execution
> purpose I put in is "test the new call to the CMU provenance service,
> by Lei, 3/20/2016."
> Thanks,
> -Lei
> urlLink: model1=GFDL_ESM2G&var1=pr&lon1=0&lon2=360&lat1=-90&lat2=90&startT=200401&endT=200412&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&image=http://cmda-test.jpl.nasa.gov:8090/static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.jpeg&data_url=http://cmda-test.jpl.nasa.gov:8090/static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.nc
> 104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /svc/twoDimMap?model=GFDL_ESM2G&var=pr&start_time=200401&end_time=200412&lon1=0&lon2=360&lat1=-90&lat2=90&months=1,2,3,4,5,6,7,8,9,10,11,12&scale=0&purpose=test%20the%20new%20call%20to%20the%20CMU%20provenance%20service,%20by%20Lei,%203/20/2016. HTTP/1.1" 200 -
> source: GFDL/ESM2G
> var: pr
> /mnt/data_2015/cmip5/gfdl/esm2g/./pr_*.nc
> retDateList: [199601, 200512]
> Wei's URL called here
> 104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /svc/time_bounds?serviceType=1&source=GFDL/ESM2G&var=pr HTTP/1.1" 200 -
> 104.33.89.198 - - [21/Mar/2016 06:41:59] "GET /static/twoDimMap/420d6b482163307c98e624e26d447f34/gfdl_esm2g_pr_200401_200412_Annual.jpeg HTTP/1.1" 200 -
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)