You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@oodt.apache.org by "Chris A. Mattmann (JIRA)" <ji...@apache.org> on 2011/02/13 01:33:04 UTC
[jira] Created: (OODT-139) PCS JAX-RS services
PCS JAX-RS services
-------------------
Key: OODT-139
URL: https://issues.apache.org/jira/browse/OODT-139
Project: OODT
Issue Type: New Feature
Components: pcs
Reporter: Chris A. Mattmann
Assignee: Chris A. Mattmann
Fix For: 0.3
Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995629#comment-12995629 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
One note: [~ahart], your patch only changed the output JSON format for /crawlers, but not for /crawlers/{cname} and not for the full report, /report. In r1071471, I added the same change for those methods.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996694#comment-12996694 ]
Chris A. Mattmann edited comment on OODT-139 at 2/19/11 4:11 AM:
-----------------------------------------------------------------
To run the test case for PedigreeResource, install the latest copy of pcs-services, and then in the background, grab the pedigree_files_and_policy.tar.gz and do the following:
# untar the policy files into your filemgr/policy directory.
# edit your filemgr/etc/filemgr.properties and point it to your new policy (if it's not put in the default place).
# dump the files/ directory somewhere on disk.
# edit the met files inside of files/\*.met and change the *FileLocation* met parameter to point to the path on disk where you dumped the files/ directory.
# restart your filemgr.
# ingest the files/met out of the files/ directory into your file manager.
After you execute the steps above you'll have products with the *InputFiles* and *OutputFiles* met fields set and you'll be able run some sample PCS traces.
was (Author: chrismattmann):
To run the test case for PedigreeResource, install the latest copy of pcs-services, and then in the background, grab the pedigree_files_and_policy.tar.gz and do the following:
# untar the policy files into your filemgr/policy directory.
# edit your filemgr/etc/filemgr.properties and point it to your new policy (if it's not put in the default place).
# dump the files/ directory somewhere on disk.
# edit the met files inside of files/*.met and change the *FileLocation* met parameter to point to the path on disk where you dumped the files/ directory.
# restart your filemgr.
# ingest the files/met out of the files/ directory into your file manager.
After you execute the steps above you'll have products with the *InputFiles* and *OutputFiles* met fields set and you'll be able run some sample PCS traces.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt, pedigree_files_and_policy.tar.gz
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996264#comment-12996264 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
Added the docs on the Pedigree service: https://cwiki.apache.org/confluence/display/OODT/OODT+REST+Services
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994821#comment-12994821 ]
Chris A. Mattmann edited comment on OODT-139 at 2/15/11 3:15 PM:
-----------------------------------------------------------------
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
h2. Just the Daemons
To see the Health Monitor report, focused on just the PCS daemons (including batch stubs), access:
{noformat}
http://host/pcs-services/services/health/report/daemon
{noformat}
Note that daemons can be further "sliced out" by adding additional REST parameters, e.g.:
h3. To see just the file manager status:
{noformat}
http://host/pcs-services/services/health/report/daemon/fm
{noformat}
h3. To see just the workflow manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/wm
{noformat}
h3. To see just the resource manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/rm
{noformat}
h3. To see just the batch stub status
{noformat}
http://host/pcs-services/services/health/report/daemon/stubs
{noformat}
was (Author: chrismattmann):
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"batch stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995134#comment-12995134 ]
Brian Foster commented on OODT-139:
-----------------------------------
NOTE: wengine is synonymous to wmgr (i.e. truck workflow manager)
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995632#comment-12995632 ]
Brian Foster edited comment on OODT-139 at 2/17/11 1:57 AM:
------------------------------------------------------------
*endpoint for viewing a workflow instance*
{noformat}
{
"workflow": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestModelId",
"modelname": "TestModelName",
"postconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPostCondModelId",
"modelname": "TestPostCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Done"
},
"timesblocked": 2
},
"preconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPreCondModelId",
"modelname": "TestPreCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Done"
},
"timesblocked": 2
},
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Done"
},
"subprocessors": [
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Done"
},
"timesblocked": 2
},
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Done"
},
"timesblocked": 2
}
],
"timesblocked": 2
}
}
{noformat}
was (Author: bfoster):
*endpoint for viewing a workflow instance*
{noformat}
{
"workflow": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestModelId",
"modelname": "TestModelName",
"postconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPostCondModelId",
"modelname": "TestPostCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
"preconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPreCondModelId",
"modelname": "TestPreCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"subprocessors": [
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
}
],
"timesblocked": 2
}
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995418#comment-12995418 ]
Paul Ramirez commented on OODT-139:
-----------------------------------
@Chris Instead of documenting the services here I think we should just move these into our Wiki. It will be easier to organize and maintain. I'll add that page soon.
@Brian Seeing your documentation helps a lot. I have an issue with the approach that is being taken given the responses that are coming from your service. It seems as though you are trying to be too generic and accomplish too much. Maps are being encoded into JSON even though there are structs that would mimic that. Why would someone need the list of columns when they can inspect the first result? Check out http://it.toolbox.com/blogs/enterprise-web-solutions/json-basics-in-javascript-32660. What also should be considered is the interpretation of the JSON by the client. Here is a sample discussion that somewhat illustrates what I'm saying http://stackoverflow.com/questions/1637334/iterating-through-parsing-json-object-via-javascript. Moreover, even if we were implementing a column service, which I don't recommend we do, why would it not just return ["InstanceId","ModelId","State","CreationDate","CurrentTasks","AliveTime"]? If you agree on the endpoints, parameters, and messages passed then both the interface and server side implementer can begin work and use that as a contract.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994870#comment-12994870 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
+1, documentation looks great!
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995389#comment-12995389 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
I think we need to take a look at the Workflow JSON response. We might be able to pair down the nesting a bit. Thanks!
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-139) PCS JAX-RS services
Posted by "Andrew Hart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Hart updated OODT-139:
-----------------------------
Attachment: OODT-139.Hart.021611.patch.txt
Patch for my previous comment regarding using array notation for all lists.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994821#comment-12994821 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"batch stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Andrew Hart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995569#comment-12995569 ]
Andrew Hart commented on OODT-139:
----------------------------------
There was a slight inconsistency in the webapp context.xml file with the way config paths were being specified (2 contained a /pcs/ segment and 1 did not). r1071436 fixes this.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-139:
-----------------------------------
Attachment: pedigree_files_and_policy.tar.gz
- example files and met files and policy to set up a test case for PedigreeResource
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt, pedigree_files_and_policy.tar.gz
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Andrew Hart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994937#comment-12994937 ]
Andrew Hart commented on OODT-139:
----------------------------------
Got this working locally, and it ROCKS. There were just a couple sticking points for me:
* the PCS core package assumes the existence of three environment variables: FILEMGR_URL, WORKFLOW_URL, and RESMGR_URL. Setting these allowed me to continue
* trunk/pcs/services/src/main/webapp/META-INF/context.xml contains two local path variables that need to be set to the physical location of the pcs-crawlers.xml and pcs-workflow-statuses.xml respectively. (Both of these files exist in trunk/pcs/core/src/main/resources/
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996692#comment-12996692 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
In r1072241, I committed the initial version of the Pedigree Resource. See the wiki for docs and response output format.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994821#comment-12994821 ]
Chris A. Mattmann edited comment on OODT-139 at 2/17/11 1:51 AM:
-----------------------------------------------------------------
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": [
{
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
],
"daemonStatus": {
"stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
h2. Just the Daemons
To see the Health Monitor report, focused on just the PCS daemons (including batch stubs), access:
{noformat}
http://host/pcs-services/services/health/report/daemon
{noformat}
Note that daemons can be further "sliced out" by adding additional REST parameters, e.g.:
h3. To see just the file manager status:
{noformat}
http://host/pcs-services/services/health/report/daemon/fm
{noformat}
h3. To see just the workflow manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/wm
{noformat}
h3. To see just the resource manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/rm
{noformat}
h3. To see just the batch stub status
{noformat}
http://host/pcs-services/services/health/report/daemon/stubs
{noformat}
h2. Just the Crawlers
To see the Health Monitor report, focused on just the PCS ingest crawlers, access:
{noformat}
http://host/pcs-services/services/health/report/crawlers
{noformat}
Crawlers can also be sliced out, similar to daemons, e.g.:
h3. Slice a crawler out by name
To see a crawler with the name Crawler1, slice it out by:
{noformat}
http://host/pcs-services/services/health/report/crawlers/Crawler1
{noformat}
h2. Just Job Processing Status
To see the Health Monitor report, focused on just the PCS workflow (job) processing status, broken down by state, access:
{noformat}
http://host/pcs-services/services/health/report/jobs
{noformat}
Job status may also be sliced out, similar to daemons and crawlers, by using the state parameter, e.g.:
h3. Slice a job processing status by state
The following will display the number of jobs that are in the QUEUED state:
{noformat}
http://host/pcs-services/services/health/report/jobs/QUEUED
{noformat}
h2. Just Ingest Processing Health
To see the Health Monitor report, focused on just the PCS ingest crawler health (with information like number of crawls and average crawl time), broken down by Crawler, access:
{noformat}
http://host/pcs-services/services/health/report/ingest
{noformat}
Ingest crawler health status can be sliced, similar to daemon status, crawler status, and the other types of information, e.g.:
h3. Slice ingest crawler health status by name
To see the ingest health status of the Crawler with the name Crawler1, access:
{noformat}
http://host/pcs-services/services/health/report/ingest/Crawler1
{noformat}
was (Author: chrismattmann):
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
h2. Just the Daemons
To see the Health Monitor report, focused on just the PCS daemons (including batch stubs), access:
{noformat}
http://host/pcs-services/services/health/report/daemon
{noformat}
Note that daemons can be further "sliced out" by adding additional REST parameters, e.g.:
h3. To see just the file manager status:
{noformat}
http://host/pcs-services/services/health/report/daemon/fm
{noformat}
h3. To see just the workflow manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/wm
{noformat}
h3. To see just the resource manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/rm
{noformat}
h3. To see just the batch stub status
{noformat}
http://host/pcs-services/services/health/report/daemon/stubs
{noformat}
h2. Just the Crawlers
To see the Health Monitor report, focused on just the PCS ingest crawlers, access:
{noformat}
http://host/pcs-services/services/health/report/crawlers
{noformat}
Crawlers can also be sliced out, similar to daemons, e.g.:
h3. Slice a crawler out by name
To see a crawler with the name Crawler1, slice it out by:
{noformat}
http://host/pcs-services/services/health/report/crawlers/Crawler1
{noformat}
h2. Just Job Processing Status
To see the Health Monitor report, focused on just the PCS workflow (job) processing status, broken down by state, access:
{noformat}
http://host/pcs-services/services/health/report/jobs
{noformat}
Job status may also be sliced out, similar to daemons and crawlers, by using the state parameter, e.g.:
h3. Slice a job processing status by state
The following will display the number of jobs that are in the QUEUED state:
{noformat}
http://host/pcs-services/services/health/report/jobs/QUEUED
{noformat}
h2. Just Ingest Processing Health
To see the Health Monitor report, focused on just the PCS ingest crawler health (with information like number of crawls and average crawl time), broken down by Crawler, access:
{noformat}
http://host/pcs-services/services/health/report/ingest
{noformat}
Ingest crawler health status can be sliced, similar to daemon status, crawler status, and the other types of information, e.g.:
h3. Slice ingest crawler health status by name
To see the ingest health status of the Crawler with the name Crawler1, access:
{noformat}
http://host/pcs-services/services/health/report/ingest/Crawler1
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-139:
-----------------------------------
Attachment: OODT-139.Mattmann.021411.2.patch.txt
- okey dok, fully functioning PCS health monitor. If there are no objections, going to commit this in next 24 hours.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995133#comment-12995133 ]
Brian Foster commented on OODT-139:
-----------------------------------
*In order to determine result's map entry keys there will be the following workflow endpoint:*
{noformat}
http://localhost:8080/cas-webservices/wengine/columns[.xml | .json]
- path params: none
- query params: none
- example:
|_ url:
http://localhost:8080/cas-webservices/wengine/columns.json
|_ json response:
{
"list": {
"value": [
"InstanceId",
"ModelId",
"State",
"CreationDate",
"CurrentTasks",
"AliveTime"
]
}
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995872#comment-12995872 ]
Paul Ramirez edited comment on OODT-139 at 2/17/11 4:14 PM:
------------------------------------------------------------
Agree with Chris on the terminology to use should be at the cas-workflow level and the root URL for the service should have "/wfmgr" and not "/wengine". While this may seem merely cosmetic it will help keep thing straight when looking at documentation and architecture diagrams.
@Brian Let's move this documentation to the Wiki page. If that is an endpoint for viewing a workflow instance there is no need for the outer "workflow:" and instead should just start with the { "completiondate": ....}. The consumer of this endpoint should know that they will be getting a workflow instance as that is the way the documentation will read. Also, what does the endpoint look like? My guess is something along the lines of /workflows/{instanceid}. In general it may make sense to return a more minimal representation of the workflow instance instead of all the level of detail you have above. For instance, maybe do something like "posconditions":["TestPostCondModelName"]. Then you could have an endpoint to get the set of postconditions (i.e. /workflows/{instanceid}/postconditions) and maybe even an endpoint for addressing a single condition (i.e. /workflows/{instanceid}/postconditions/{modelname}). The same could be said for preconditions and subprocessors as its not very likely that you are going to need all this information unless you have a specific use case in mind. Thanks for pushing this stuff out there it really helps to see where its going on early on.
was (Author: pramirez):
Let's move this documentation to the Wiki page. If that is an endpoint for viewing a workflow instance there is no need for the outer "workflow:" and instead should just start with the { "completiondate": ....}. The consumer of this endpoint should know that they will be getting a workflow instance as that is the way the documentation will read. Also, what does the endpoint look like? My guess is something along the lines of /workflows/{instanceid}. In general it may make sense to return a more minimal representation of the workflow instance instead of all the level of detail you have above. For instance, maybe do something like "posconditions":["TestPostCondModelName"]. Then you could have an endpoint to get the set of postconditions (i.e. /workflows/{instanceid}/postconditions) and maybe even an endpoint for addressing a single condition (i.e. /workflows/{instanceid}/postconditions/{modelname}). The same could be said for preconditions and subprocessors as its not very likely that you are going to need all this information unless you have a specific use case in mind.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995045#comment-12995045 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
- Committed the HealthMonitor in r1071097, r1071098, and r1071099. Will work on the PedigreeResource next.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995872#comment-12995872 ]
Paul Ramirez commented on OODT-139:
-----------------------------------
Let's move this documentation to the Wiki page. If that is an endpoint for viewing a workflow instance there is no need for the outer "workflow:" and instead should just start with the { "completiondate": ....}. The consumer of this endpoint should know that they will be getting a workflow instance as that is the way the documentation will read. Also, what does the endpoint look like? My guess is something along the lines of /workflows/{instanceid}. In general it may make sense to return a more minimal representation of the workflow instance instead of all the level of detail you have above. For instance, maybe do something like "posconditions":["TestPostCondModelName"]. Then you could have an endpoint to get the set of postconditions (i.e. /workflows/{instanceid}/postconditions) and maybe even an endpoint for addressing a single condition (i.e. /workflows/{instanceid}/postconditions/{modelname}). The same could be said for preconditions and subprocessors as its not very likely that you are going to need all this information unless you have a specific use case in mind.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-139:
-----------------------------------
Attachment: OODT-139.Mattmann.021311.patch.txt
- patch showing the HealthResource and associated config. Accessible via:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
and
{noformat}
http://host/pcs-services/services/health/report/daemon/{name}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996694#comment-12996694 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
To run the test case for PedigreeResource, install the latest copy of pcs-services, and then in the background, grab the pedigree_files_and_policy.tar.gz and do the following:
#untar the policy files into your filemgr/policy directory.
#edit your filemgr/etc/filemgr.properties and point it to your new policy (if it's not put in the default place).
#dump the files/ directory somewhere on disk.
#edit the met files inside of files/*.met and change the *FileLocation* met parameter to point to the path on disk where you dumped the files/ directory.
#restart your filemgr.
#ingest the files/met out of the files/ directory into your file manager.
After you execute the steps above you'll have products with the *InputFiles* and *OutputFiles* met fields set and you'll be able run some sample PCS traces.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt, pedigree_files_and_policy.tar.gz
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann updated OODT-139:
-----------------------------------
Attachment: OODT-139.Mattmann.021411.patch.txt
- updated patch with more fleshed out health monitor that includes
# generatedTime
# damoneStatus {fm|wm|rm}
# crawlerStatus
# latest files
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994858#comment-12994858 ]
Paul Ramirez commented on OODT-139:
-----------------------------------
Suggestion to update documentation. Let me know if you like the format and I'll do it for the rest of the ones listed above.
h3. Slice a job processing status by state
----
h4.Endpoint:
{noformat}
http://host/pcs-services/services/health/report/jobs/{state}
{noformat}
h4.HTTP Request
*Method:* GET
*Parameters:*
||Name||Placement||Type||Description||
|state|path|Enum|Values range from ["STARTED", "FINISHED", "PAUSED", "RSUBMIT", "QUEUED"]. There may be extended states but these are a list of the basic ones. Extended ones would be passed in the same manner.|
h4.HTTP Response
h5.Valid
Status: 200 OK
*JSON*
{noformat}
{
"report":{
"generated":"2011-02-15T08:09:24.225-0800",
"jobHealth":[
{
"state":"QUEUED",
"numJobs":0
}
]
}
}
{noformat}
h4.Example(s):
To display the number of jobs that are in the QUEUED state
{noformat}
http://host/pcs-services/services/health/report/jobs/QUEUED
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995143#comment-12995143 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
Thanks for your documentation Brian!
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995624#comment-12995624 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
I applied Andrew's patch in r1071466. Thanks, Andrew!
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996214#comment-12996214 ]
Brian Foster commented on OODT-139:
-----------------------------------
*UPDATES TO ENDPOINTS MENTIONED ABOVE*
*Page of Workflows*
URL: http://localhost:8080/cas-webservices/wmgr/workflows.json?size=2
JSON:
{noformat}
{
"number": 1,
"size": 2,
"workflows": [
{
"alivetime": 20000,
"creationdate": "2011-02-17T18:27:50.288-08:00",
"currenttasks": [
"Test1Task",
"Test2Task"
],
"instanceid": "inst-0",
"workflowid": "TestWorkflow"
},
{
"alivetime": 20000,
"creationdate": "2011-02-17T18:27:50.288-08:00",
"currenttasks": [
"Test1Task",
"Test2Task"
],
"instanceid": "inst-1",
"workflowid": "TestWorkflow"
}
],
"hits": 100
}
{noformat}
*Get Workflow By InstanceId*
URL: http://localhost:8080/cas-webservices/wmgr/workflows/inst-1.json
JSON:
{noformat}
{
"completiondate": "2011-02-17T18:33:56.757-08:00",
"creationdate": "2011-02-17T18:33:56.757-08:00",
"executiondate": "2011-02-17T18:33:56.757-08:00",
"instanceid": "inst-1",
"postconditions": {
"completiondate": "2011-02-17T18:33:56.757-08:00",
"creationdate": "2011-02-17T18:33:56.757-08:00",
"executiondate": "2011-02-17T18:33:56.757-08:00",
"instanceid": "inst-1",
"priority": 5,
"readydate": "2011-02-17T18:33:56.757-08:00",
"state": {
"date": "2011-02-17T18:33:56.757-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2,
"workflowid": "TestPostCondWorkflowId",
"workflowname": "TestPostCondWorkflowName"
},
"preconditions": {
"completiondate": "2011-02-17T18:33:56.757-08:00",
"creationdate": "2011-02-17T18:33:56.757-08:00",
"executiondate": "2011-02-17T18:33:56.757-08:00",
"instanceid": "inst-1",
"priority": 5,
"readydate": "2011-02-17T18:33:56.757-08:00",
"state": {
"date": "2011-02-17T18:33:56.757-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2,
"workflowid": "TestPreCondWorkflowId",
"workflowname": "TestPreCondWorkflowName"
},
"priority": 5,
"readydate": "2011-02-17T18:33:56.757-08:00",
"state": {
"date": "2011-02-17T18:33:56.757-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"subprocessors": [
{
"completiondate": "2011-02-17T18:33:56.757-08:00",
"creationdate": "2011-02-17T18:33:56.757-08:00",
"executiondate": "2011-02-17T18:33:56.757-08:00",
"instanceid": "inst-1",
"priority": 5,
"readydate": "2011-02-17T18:33:56.757-08:00",
"state": {
"date": "2011-02-17T18:33:56.757-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2,
"workflowid": "TestSubProcWorkflowId",
"workflowname": "TestSubProcWorkflowName"
},
{
"completiondate": "2011-02-17T18:33:56.757-08:00",
"creationdate": "2011-02-17T18:33:56.757-08:00",
"executiondate": "2011-02-17T18:33:56.757-08:00",
"instanceid": "inst-1",
"priority": 5,
"readydate": "2011-02-17T18:33:56.757-08:00",
"state": {
"date": "2011-02-17T18:33:56.757-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2,
"workflowid": "TestSubProcWorkflowId",
"workflowname": "TestSubProcWorkflowName"
}
],
"timesblocked": 2,
"workflowid": "TestWorkflowId",
"workflowname": "TestWorkflowName"
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Paul Ramirez (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995617#comment-12995617 ]
Paul Ramirez commented on OODT-139:
-----------------------------------
Created a that Wiki page to start to capture our service documentation https://cwiki.apache.org/confluence/display/OODT/OODT+REST+Services. I would have taken some of the documentation from the comments here but I can't edit peoples comments to pilfer the formatting they have already done.
@Brian Yep looks great. Reading that JSON the intent of the service seems to come through in the response.
@Andrew I like the update to the JSON.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995128#comment-12995128 ]
Brian Foster commented on OODT-139:
-----------------------------------
*Page through workflows endpoint:*
{noformat}
http://localhost:8080/cas-webservices/wengine/workflows[.xml | .json]
- path params: none
- query params:
|_ size : the page size
|_ page : the page number
|_ state : filter workflows to given state
|_ category : filter workflows to given category
|_ modelid : filter workflows to given model id
|_ metadata : filter workflows to given metadata (format <key>:<val>; specify a list of values: metadata=key1:val1&metadata=key1:val2)
- example:
|_ url :
http://localhost:8080/cas-webservices/wengine/workflows.json?page=1&size=2&state=Executing&metadata=NominalDate:2008-01-20&metadata=NominalDate:2008-01-21&metadata=CollectionLabel:TestRun
|_ json responce:
{
"page": {
"number": 1,
"size": 2,
"results": [
{
"map": {
"entries": [
{
"key": "ModelId",
"values": "TestWorkflow"
},
{
"key": "State",
"values": "Executing"
},
{
"key": "InstanceId",
"values": "inst-0"
},
{
"key": "AliveTime",
"values": "20 hours"
},
{
"key": "CreationDate",
"values": "2008-01-20T00:00:00.000Z"
},
{
"key": "CurrentTasks",
"values": [
"Test1Task",
"Test2Task"
]
}
]
},
"id": "inst-0"
},
{
"map": {
"entries": [
{
"key": "ModelId",
"values": "TestWorkflow"
},
{
"key": "State",
"values": "Executing"
},
{
"key": "InstanceId",
"values": "inst-1"
},
{
"key": "AliveTime",
"values": "20 hours"
},
{
"key": "CreationDate",
"values": "2008-01-20T00:00:00.000Z"
},
{
"key": "CurrentTasks",
"values": [
"Test1Task",
"Test2Task"
]
}
]
},
"id": "inst-1"
}
],
"hits": 100
}
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris A. Mattmann resolved OODT-139.
------------------------------------
Resolution: Fixed
I'm marking this as closed. I recorded the changelog entry in r1080473. Thanks to Paul, Brian and Andrew for comments/review. Documentation is up on the wiki for the health service and for the pedigree service. Feedback welcome!
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt, pedigree_files_and_policy.tar.gz
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995632#comment-12995632 ]
Brian Foster commented on OODT-139:
-----------------------------------
*endpoint for viewing a workflow instance*
{noformat}
{
"workflow": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestModelId",
"modelname": "TestModelName",
"postconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPostCondModelId",
"modelname": "TestPostCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
"preconditions": {
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestPreCondModelId",
"modelname": "TestPreCondModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"subprocessors": [
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
},
{
"completiondate": "2011-02-16T17:54:32.544-08:00",
"creationdate": "2011-02-16T17:54:32.544-08:00",
"executiondate": "2011-02-16T17:54:32.544-08:00",
"instanceid": "inst-1",
"modelid": "TestSubProcModelId",
"modelname": "TestSubProcModelName",
"priority": 5,
"readydate": "2011-02-16T17:54:32.544-08:00",
"state": {
"date": "2011-02-16T17:54:32.544-08:00",
"message": "This is a state's message",
"state": "Executing"
},
"timesblocked": 2
}
],
"timesblocked": 2
}
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Andrew Hart (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995605#comment-12995605 ]
Andrew Hart commented on OODT-139:
----------------------------------
the HealthResource service at /health/report/crawlers currently uses JSON object notation to represent a list of crawler statuses. In other parts of the service, though, (e.g.: listing latest files, job statuses), lists are expressed using array notation. It might be more consistent to use array notation here too. I'm attaching a patch (OODT-139.Hart.20110216.patch.txt) which will implement this. It changes the output of:
/health/report/crawlers
{noformat}
{"report": {"crawlerStatus":{
"Crawler1" : {"status":"DOWN","crawlerName":"Crawler1","crawlerPort":"9020","url":"localhost"}},"generated":"2011-02-16T06:18:49.263-0800"},
"Crawler2" : {"status":"DOWN","crawlerName":"Crawler2","crawlerPort":"9021","url":"localhost"}},"generated":"2011-02-16T06:18:49.263-0800"},
"Crawler3" : {"status":"DOWN","crawlerName":"Crawler3","crawlerPort":"9022","url":"localhost"}},"generated":"2011-02-16T06:18:49.263-0800"}
}
{noformat}
to:
/health/report/crawlers
{noformat}
{"report": {"crawlerStatus": [
{"status":"DOWN","crawlerName":"Crawler1","crawlerPort":"9020","url":"localhost"},
{"status":"DOWN","crawlerName":"Crawler2","crawlerPort":"9021","url":"localhost"},
{"status":"DOWN","crawlerName":"Crawler3","crawlerPort":"9022","url":"localhost"},
],"generated":"2011-02-16T06:18:49.263-0800"}
}
{noformat}
Note that this change does not effect the functionality of /health/report/crawlers/CrawlerName.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12996694#comment-12996694 ]
Chris A. Mattmann edited comment on OODT-139 at 2/19/11 4:10 AM:
-----------------------------------------------------------------
To run the test case for PedigreeResource, install the latest copy of pcs-services, and then in the background, grab the pedigree_files_and_policy.tar.gz and do the following:
# untar the policy files into your filemgr/policy directory.
# edit your filemgr/etc/filemgr.properties and point it to your new policy (if it's not put in the default place).
# dump the files/ directory somewhere on disk.
# edit the met files inside of files/*.met and change the *FileLocation* met parameter to point to the path on disk where you dumped the files/ directory.
# restart your filemgr.
# ingest the files/met out of the files/ directory into your file manager.
After you execute the steps above you'll have products with the *InputFiles* and *OutputFiles* met fields set and you'll be able run some sample PCS traces.
was (Author: chrismattmann):
To run the test case for PedigreeResource, install the latest copy of pcs-services, and then in the background, grab the pedigree_files_and_policy.tar.gz and do the following:
#untar the policy files into your filemgr/policy directory.
#edit your filemgr/etc/filemgr.properties and point it to your new policy (if it's not put in the default place).
#dump the files/ directory somewhere on disk.
#edit the met files inside of files/*.met and change the *FileLocation* met parameter to point to the path on disk where you dumped the files/ directory.
#restart your filemgr.
#ingest the files/met out of the files/ directory into your file manager.
After you execute the steps above you'll have products with the *InputFiles* and *OutputFiles* met fields set and you'll be able run some sample PCS traces.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt, pedigree_files_and_policy.tar.gz
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995638#comment-12995638 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
Hey Brian, can we make the workflow JSON use existing cas-workflow terminology, rather than wengine? So, instead of modelId, can we use workflowId? Also modelname should be workflowName. I'd like to help you with the key value names b/c I'd love to use this as I merge wengine into cas-workflow.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Brian Foster (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995599#comment-12995599 ]
Brian Foster commented on OODT-139:
-----------------------------------
hey paul better?
{noformat}
{
"page": {
"number": 1,
"size": 2,
"workflows": [
{
"alivetime": 20000,
"creationdate": "2011-02-16T16:12:18.198-08:00",
"currenttasks": [
"Test1Task",
"Test2Task"
],
"instanceid": "inst-0",
"modelid": "TestWorkflow"
},
{
"alivetime": 20000,
"creationdate": "2011-02-16T16:12:18.198-08:00",
"currenttasks": [
"Test1Task",
"Test2Task"
],
"instanceid": "inst-1",
"modelid": "TestWorkflow"
}
],
"hits": 100
}
}
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994821#comment-12994821 ]
Chris A. Mattmann edited comment on OODT-139 at 2/15/11 3:00 PM:
-----------------------------------------------------------------
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"batch stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
was (Author: chrismattmann):
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"batch stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12994821#comment-12994821 ]
Chris A. Mattmann edited comment on OODT-139 at 2/15/11 3:25 PM:
-----------------------------------------------------------------
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
h2. Just the Daemons
To see the Health Monitor report, focused on just the PCS daemons (including batch stubs), access:
{noformat}
http://host/pcs-services/services/health/report/daemon
{noformat}
Note that daemons can be further "sliced out" by adding additional REST parameters, e.g.:
h3. To see just the file manager status:
{noformat}
http://host/pcs-services/services/health/report/daemon/fm
{noformat}
h3. To see just the workflow manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/wm
{noformat}
h3. To see just the resource manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/rm
{noformat}
h3. To see just the batch stub status
{noformat}
http://host/pcs-services/services/health/report/daemon/stubs
{noformat}
h2. Just the Crawlers
To see the Health Monitor report, focused on just the PCS ingest crawlers, access:
{noformat}
http://host/pcs-services/services/health/report/crawlers
{noformat}
Crawlers can also be sliced out, similar to daemons, e.g.:
h3. Slice a crawler out by name
To see a crawler with the name Crawler1, slice it out by:
{noformat}
http://host/pcs-services/services/health/report/crawlers/Crawler1
{noformat}
h2. Just Job Processing Status
To see the Health Monitor report, focused on just the PCS workflow (job) processing status, broken down by state, access:
{noformat}
http://host/pcs-services/services/health/report/jobs
{noformat}
Job status may also be sliced out, similar to daemons and crawlers, by using the state parameter, e.g.:
h3. Slice a job processing status by state
The following will display the number of jobs that are in the QUEUED state:
{noformat}
http://host/pcs-services/services/health/report/jobs/QUEUED
{noformat}
h2. Just Ingest Processing Health
To see the Health Monitor report, focused on just the PCS ingest crawler health (with information like number of crawls and average crawl time), broken down by Crawler, access:
{noformat}
http://host/pcs-services/services/health/report/ingest
{noformat}
Ingest crawler health status can be sliced, similar to daemon status, crawler status, and the other types of information, e.g.:
h3. Slice ingest crawler health status by name
To see the ingest health status of the Crawler with the name Crawler1, access:
{noformat}
http://host/pcs-services/services/health/report/ingest/Crawler1
{noformat}
was (Author: chrismattmann):
I just wanted to provide some documentation on how to call the Health Monitor service.
h2. JSON output format
All calls to the Health Monitor REST service provide the following JSON output:
{noformat}
{
"report": {
"crawlerStatus": {
"Crawler1": {
"crawlerName": "Crawler1",
"crawlerPort": "9020",
"status": "UP",
"url": "localhost"
}
},
"daemonStatus": {
"stubs": [
{
"daemon": "batch stub",
"status": "UP",
"url": "http://localhost:2001"
}
],
"fm": {
"daemon": "File Manager",
"status": "UP",
"url": "http://localhost:9000"
},
"rm": {
"daemon": "Resource Manager",
"status": "UP",
"url": "http://localhost:9002"
},
"wm": {
"daemon": "Workflow Manager",
"status": "UP",
"url": "http://localhost:9001"
}
},
"generated": "2011-02-15T06:57:07.591-0800",
"ingestHealth": [
{
"avgCrawlTime": 132.78640211640212,
"crawler": "Crawler1",
"numCrawls": 189
}
],
"jobHealth": [
{
"numJobs": 0,
"state": "QUEUED"
},
{
"numJobs": 0,
"state": "RSUBMIT"
},
{
"numJobs": 0,
"state": "BUILDING CONFIG FILE"
},
{
"numJobs": 0,
"state": "PGE EXEC"
},
{
"numJobs": 0,
"state": "CRAWLING"
},
{
"numJobs": 0,
"state": "STAGING INPUT"
},
{
"numJobs": 7,
"state": "FINISHED"
},
{
"numJobs": 0,
"state": "STARTED"
},
{
"numJobs": 0,
"state": "PAUSED"
}
],
"latestFiles": {
"files": [
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:19:21.126-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:08:10.198-08:00"
},
{
"filepath": "/Users/mattmann/files/foo.bar/foo.bar",
"receivedTime": "2011-01-22T15:06:03.659-08:00"
},
{
"filepath": "/Users/mattmann/files/blah.txt/blah.txt",
"receivedTime": "2011-01-21T21:56:03.922-08:00"
}
],
"topN": 20
}
}
}
{noformat}
The below REST-ful service descriptions show how to slice out different parts of the report. All reports at least have the *generatedTime* attribute, and then some combination of *daemonStatus*, and/or *crawlerStatus*, and (if the file manager is running) *latestFiles*, and if the workflow manager is running *jobHealth*, and if the crawlers are running *ingestHealth*.
h2. Full Report
To see the full Health Monitor report, access:
{noformat}
http://host/pcs-services/services/health/report
{noformat}
h2. Just the Daemons
To see the Health Monitor report, focused on just the PCS daemons (including batch stubs), access:
{noformat}
http://host/pcs-services/services/health/report/daemon
{noformat}
Note that daemons can be further "sliced out" by adding additional REST parameters, e.g.:
h3. To see just the file manager status:
{noformat}
http://host/pcs-services/services/health/report/daemon/fm
{noformat}
h3. To see just the workflow manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/wm
{noformat}
h3. To see just the resource manager status
{noformat}
http://host/pcs-services/services/health/report/daemon/rm
{noformat}
h3. To see just the batch stub status
{noformat}
http://host/pcs-services/services/health/report/daemon/stubs
{noformat}
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (OODT-139) PCS JAX-RS services
Posted by "Chris A. Mattmann (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/OODT-139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12995623#comment-12995623 ]
Chris A. Mattmann commented on OODT-139:
----------------------------------------
[~ahart] Thanks, this looks great! I'll go ahead and apply your patch shortly.
[~pramirez] Check out: https://cwiki.apache.org/confluence/display/OODT/OODT+REST+Services I've added my documentation to the wiki page
[~bfoster] +1 to [~pramirez]'s comments, your JSON output looks good. Let's start to collect your workflow services into a new issue b/c OODT-139 is really the PCS higher level REST tools.
> PCS JAX-RS services
> -------------------
>
> Key: OODT-139
> URL: https://issues.apache.org/jira/browse/OODT-139
> Project: OODT
> Issue Type: New Feature
> Components: pcs
> Reporter: Chris A. Mattmann
> Assignee: Chris A. Mattmann
> Labels: jaxrs, jsr311, pcs
> Fix For: 0.3
>
> Attachments: OODT-139.Hart.021611.patch.txt, OODT-139.Mattmann.021311.patch.txt, OODT-139.Mattmann.021411.2.patch.txt, OODT-139.Mattmann.021411.patch.txt
>
>
> Now that the PCS core is contributed, it would be great to not just have command line versions and APIs for tools like pcs_stat, pcstrace and pcs_ll. We should expose them over JAX-RS so that webapps can easily take advantage of PCS services.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira