You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@stanbol.apache.org by rw...@apache.org on 2012/02/17 11:29:45 UTC
svn commit: r1245375 -
/incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext
Author: rwesten
Date: Fri Feb 17 10:29:45 2012
New Revision: 1245375
URL: http://svn.apache.org/viewvc?rev=1245375&view=rev
Log:
Documentation for the enhancer RESTful API
Added:
incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext
Added: incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext
URL: http://svn.apache.org/viewvc/incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext?rev=1245375&view=auto
==============================================================================
--- incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext (added)
+++ incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/enhancerrest.mdtext Fri Feb 17 10:29:45 2012
@@ -0,0 +1,305 @@
+title: Stanbol Enhancer RESTful API
+
+
+<p>The RESTful service endpoint provided by the Stanbol Enhancer is a stateless interface that allows the caller to submit content and get the resulting enhancements formatted as RDF at once without storing anything on the server-side. More advanced options also allow to parse pre-existing metadata, parse and request alternate content versions and additional metadata created by the Enhancer or specific Enhancement Engines.</p>
+
+The here described RESTful interface is provided on several Endpoints
+
+* __'/enhancer':__ The main Endpoint of the Stanbol Enhancer. Parsed content will get enhanced by using the default enhancement chain.
+* __'/enhancer/chain/{chain-name}'__: The Stanbol Enhancer supports the configuration of multiple [Enhancement Chains](chains). Users can lookup active chains by requests to the 'enhancer/chain' endpoint.
+* __'/engines':__ Same as '/enhancer' this ensures backward compatibility to older Stanbol versions.
+
+## Enhancement Request
+
+This sections describes how to parse Content to the Stanbol Enhancer that gets than analyzed. Results are sent back in the form of a serialized RDF graph.
+
+The content to analyze should be sent in a POST request with the mimetype specified in
+the <code>Content-type</code> header. The response will hold the RDF enhancement serialized in the format specified in the <code>Accept</code> header:
+
+ :::bash
+ curl -X POST -H "Accept: text/turtle" -H "Content-type: text/plain" \
+ --data "John Smith was born in London." ${it.serviceUrl}
+
+The list of mimetypes accepted as inputs depends on the deployed engines. By default most Enhancement Engines can only process plain text content. However EnhancementEngines like [Metaxa](engines/metaxaengine.html) can be used to create 'text/plain' versions of parsed content. This allows also to enhance contents with mime types such as html, pdf and MS office documents (see the Metaxa documentation for details)
+
+Stanbol enhancer is able to serialize the response in the following RDF formats:
+
+ :::text
+ application/json (JSON-LD)
+ application/rdf+xml (RDF/XML)
+ application/rdf+json (RDF/JSON)
+ text/turtle (Turtle)
+ text/rdf+nt (N-TRIPLES)
+
+### Additional supported QueryParameters:
+
+* __uri={content-item-uri}:__ By default the URI of the content item being enhanced is a local, non de-referencable URI automatically built out of a hash digest of the binary content. Sometimes it might be helpful to provide the URI of the [ContentItem](contentitem.html) to be used in the enhancements RDF graph.
+* __executionmetadata=true/false:__ Allows the include of [execution metadata](executionmetadata.html) in the enhancement metadata of the response. Such data include also the [execution plan](chains/executionplan.html) used to enhance the parsed content. This information is typically only useful to clients that want to know how the parsed content was processed by the enhancer. NOTE that the execution metadata can also be requested by using the multi-part content item API described below.
+
+The following example shows how to send an enhancement request with a
+custom content item URI that will include the execution metadata in the
+response.
+
+ :::bash
+ curl -X POST -H "Accept: text/turtle" -H "Content-type: text/plain" \
+ --data "John Smith was born in London." \
+ "${it.serviceUrl}?uri=urn:fise-example-content-item&executionmetadata=true"
+
+
+## Multi-part ContentItem support
+
+The multi-part ContentItem extensions to the RESTful API (introduced by [STANBOL-481](https://issues.apache.org/jira/browse/STANBOL-481)) are considered an advanced usage of the Stanbol Enhancer.
+
+Users will want to use this extensions if they need to:
+
+* parse multiple versions of the content: Most CMS already do have support for converting content to plain text. This API allows to parse both the original AND multiple transcoded versions of the content to the Enhancer.
+* parse pre-existing metadata: Typically CMS do have already some metadata about content parsed to the Stanbol Enhancer (e.g. User provided Tags, Categories â¦). The multi-part extensions do allow to parse such data in addition to the content.
+* request transcoded versions of the parsed content: This API extensions allows to include transcoded (e.g. the 'plain/text') version of parsed content in the response. It also allows requests that directly returns transcoded content by omitting extracted metadata.
+* request additional metadata that are normally not included within the metadata of the Enhancement response: This can to request the [execution metadata](executionmetadata.html) in an own RDF graph, but it can also be used to request metadata of specific enhancement engines (TODO: add example)
+
+
+### QueryParameters
+
+The following QueryParameters are defined by the multi-part content item extension:
+
+* __outputContentType=[mediaType]:__ Allows to specify the Mimetypes of content included within the Response of the Stanbol Enhancer. This parameter supports wild cards (e.g. '*' ... all, 'text/*'' ... all text versions, 'text/plain' ... only the plain text version). This parameter can be used multiple times.
+
+ Responses to requests with this parameter will be encoded as <code>multipart/from-data</code>. If the "Accept" header of the request is not compatible to <code>multipart/from-data</code> it is assumed as a <code>400 BAD_REQUEST</code>. For details see the documentation of the [Multipart MIME format for ContentItems](contentitem.html#multipart_mime_serialization).
+
+* __omitParsed=[true/false]:__ Makes only sense in combination with the <code>outputContentType</code> parameter. This allows to exclude all content included in the request from the response. A typical combination is <code>outputContentType=*/*&omitParsed=true</code>. The default value of this parameter is <code>false</code>
+
+* __outputContentPart=[uri/'*']:__ This parameter allows to explicitly include content parts with a specific URI in the response. Currently this only supports [ContentParts](contentitem.html#content_parts) that are stored as RDF graphs.
+
+ Responses to requests with this parameter will be encoded as <code>multipart/from-data</code>. If the "Accept" header of the request is not compatible to <code>multipart/from-data</code> it is assumed as a <code>400 BAD_REQUEST</code>. The selected content parts will be included as MIME parts in the returned [Multipart MIME formated ContentItems](contentitem.html#multipart_mime_serialization). The URI of the part will be used as name. Such parts will be added after the "metadata" and the "content" (if present).
+
+* __omitMetadata=[true/false]:__ This allows to enable/disable the inclusion of the metadata in the response. The default is <code>false</code>.
+
+ Typically <code>omitMetadata=true</code> is used when users want to use the Stanbol Enhancer just to get one or more ContentParts as an response. Note that
+Requests that use an <code>Accept: {mimeType}</code> header AND <code>omitMetadata=true</code> will directly return the content verison of <code>{mimeType}</code> and NOT wrap the result as <code>multipart/from-data</code>. See also the example further down this documentation.
+
+* __rdfFormat=[rdfMimeType]:__ This allows for requests that result in <code>multipart/from-data</code> encoded responses to specify the used RDF serialization format. Supported formats and defaults are the same as for normal Enhancer Requests.
+
+### Parsing multiple ContentParts
+
+Requests to the Stanbol Enahcer with the <code>Content-Type: multipart/from-data</code> are considered to contain a ContentItem serialized as MultiPart MIME. The exact specification of the [MultiPart MIME format for ContentItems](contentitem.html#multipart_mime_serialization) is provided by the documentation of the ContentItem.
+
+The combination of <code>multipart/from-data</code> encoded requests with QueryParameters as described above allow for the usage of [MultiPart MIME format for ContentItems](contentitem.html#multipart_mime_serialization) for both request and resonse.
+
+
+### Example usages of the multi-part content item RESTful API extensions
+
+The following examples show some typical usages of the multi-part content item RESTful API. Note that for better readability the values of the query parameters are
+not URLEncoded.
+
+Return Metadata and transformed Content versions
+
+ :::bash
+ curl -v -X POST -H "Accept: multipart/from-data" \
+ -H "Content-type: text/html; charset=UTF-8" \
+ --data "<html><body><p>John Smith was born in London.</p></body></html>" \
+ "${it.serviceUrl}?outputContent=*/*&omitParsed=true&rdfFormat=application/rdf+xml"
+
+__Example 1: Return metadata and content__
+
+This will result in an Response with the mime type <code>"Content-Type: multipart/from-data; charset=UTF-8; boundary=contentItem"</code> and the Metadata as well as the plain text version of the parsed HTML document as content.
+
+ :::text
+ --contentItem
+ Content-Disposition: form-data; name="metadata"; filename="urn:content-item-sha1-76e44d4b51c626bbed38ce88370be88702de9341"
+ Content-Type: application/rdf+xml; charset=UTF-8;
+ Content-Transfer-Encoding: 8bit
+
+ <rdf:RDF
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
+ [..the metadata formatted as RDF+XML..]
+ </rdf:RDF>
+
+ --contentItem
+ Content-Disposition: form-data; name="content"
+ Content-Type: multipart/alternate; boundary=contentParts; charset=UTF-8
+ Content-Transfer-Encoding: 8bit
+
+ --contentParts
+ Content-Disposition: form-data; name="urn:metaxa:plain-text:2daba9dc-21f6-7ea1-70dd-a2b0d5c6cd08"
+ Content-Type: text/plain; charset=UTF-8
+ Content-Transfer-Encoding: 8bit
+
+ John Smith was born in London.
+ --contentParts--
+
+ --contentItem--
+
+__Example 2: Directly return the plain text version of parsed content__
+
+This shows how the Apache Stanbol Enhancer can be used to transcode parsed content.
+
+ curl -v -X POST -H "Accept: text/plain" \
+ -H "Content-type: text/html; charset=UTF-8" \
+ --data "<html><body><p>John Smith was born in London.</p></body></html>" \
+ "${it.serviceUrl}?omitMetadata=true"
+
+The response will use <code>Content-Type: text/plain</code> and contain the string
+
+ :::text
+ John Smith was born in London.
+
+To make this work the requested [Enhancement Chain](chains) will need to include an engine (e.g. [Metaxa](engines/metaxaengine.html)) that supports transcoding the parsed content. Not that because the metadata are omitted by responses to such requests it is also recommended to configure/use a chain that does no further processing on the transcoded content.
+
+__Example 3: Parse multiple content versions__
+
+This example will use the "httpmime" part of the Apache commons httpcomponents to create the Multipart MIME sent to the Stanbol enhancer.
+
+ :::xml
+ <dependency>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>httpmime</artifactId>
+ <version>4.1.2</version>
+ </dependency>
+
+The created Multipart MIME content MUST follow the specifications as defined by the [MultiPart MIME format for ContentItems](contentitem.html#multipart_mime_serialization).
+
+ :::java
+ InputStream wordIn; //The MS Word version of the Content
+ InputStream plainIn; //The plain text version of the Content
+ HttpClient httpClient; //The client used to execute the request
+
+ //create the multipart/from-data container for the ContentItem
+ //MultipartEntity also implements HttpEntity
+ MultipartEntity contentItem = new MultipartEntity(null, null ,UTF8);
+ //The multipart/alternate container for the contents
+ HttpMultipart content = new HttpMultipart("alternate", UTF8 ,"contentParts");
+
+ //now add the container for the content to the content item container
+ contentItem.addPart(
+ "content", //the name MUST BE "content"!
+ new MultipartContentBody(content));
+
+ //now add the MS word content at the first location
+ //this will make it the "original" content
+ content.addBodyPart(new FormBodyPart(
+ "http://www.example.com/example.docx", //the id of the content part
+ new InputStreamBody(
+ wordIn,
+ "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
+ "example.docx")));
+
+ //now add the alternate plain text version
+ content.addBodyPart(new FormBodyPart(
+ "http://www.example.com/example.docx", //the id of the content part
+ new StringBody( //use a StringBody to avoid binary encoding for text
+ IOUtils.toString(plainIn), //apache commons IO utility
+ "text/plain",
+ Charset.forName("UTF-8"))));
+
+ //now we are ready to create and execute the POST request to the
+ //Stanbol Enhancer
+ HttpPost request = new HttpPost("http://localhost:8080/enhancer");
+ request.setEntity(contentItem);
+ request.setHeader("Accept","application/rdf+xml);
+ Response response = httpClient.execute(request);
+
+
+Note that for such requests [Metaxa](engines/metaxaengine.html) will still try to extract metadata of the parsed MS Word document, but all other engines will use the plain text version as parsed by the request for processing.
+
+__Example 4: Parse existing free text annotations__
+
+This example shows how the multi-part content item API can be used to parse already existing tags for an parsed content to the Stanbol Enhancer. For this example it is important to understand that parsed metadata need to confirm to the Stanbol Enhancement Structure. Because of that this example consist of two main steps:
+
+1. Convert user tags to TextAnnotations
+2. Send existing Metadata along with the Content to the Stanbol Enhancer
+
+Also note that the code snippets will uses utilities provided by the "org.apache.stannbol.enhancer.servicesapi" module. As RDF framework Clerezza is used. Both dependencies are easily replaceable.
+
+First lets have a look at the required information
+
+ :::java
+ MGraph graph; //the RDF graph to store the metadata
+ UriRef ciUri; //the URI for the contentItem
+ String tag; // user provided tag
+ UriRef tagType; //the type of the Tag
+
+Reagrding the tag type: Stanbol natively supports the following types
+
+* __Person__ (http://dbpedia.org/ontology/Person)
+* __Organization__ (http://dbpedia.org/ontology/Organisation): NOTE the British spelling
+* __Place__ (http://dbpedia.org/ontology/Place)
+
+The processing of parsed tags that use other or no type depends on the used [enhancement engines](engines) and there configurations. Especially the configuration of used the [Named Entity Tagging Engine](engines/namedentitytaggingengine.html)s is important in that respect.
+
+ :::java
+ Resource user; //the user that has created the tag (optional)
+ //in case of an name just use a literal
+ user = new PlainListeral("Rudolf Huber");
+ //in case users have assigned URIs
+ user = new UriRef("http://my.cms.org/users/rudof.huber");
+
+Now we can convert the information to TextAnnoations
+
+ :::java
+ //first create a URI for the text annotation. Here we use a random URN
+ //If you can create a meaningful URI this would be better!
+ UriRef ta = new UriRef("urn:user-annotation:"+EnhancementEngineHelper.randomUUID());
+ //The the 'rdf:type's
+ graph.add(new TripleImpl(ta, RDF.type, TechnicalClasses.ENHANCER_TEXTANNOTATION));
+ graph.add(new TripleImpl(ta, RDF.type, TechnicalClasses.ENHANCER_ENHANCEMENT));
+
+ //this TextAnnotation is about the ContentItem
+ graph.add(new TripleImpl(ta, Properties.ENHANCER_EXTRACTED_FROM, ciUri));
+ //if the Tag uses a type add it
+ if(tagType != null){
+ graph.add(new TripleImpl(ta, Properties.DC_TYPE, tagType));
+ }
+ //add the value of the tag
+ graph.add(new TripleImpl(ta, Properties.ENHANCER_SELECTED_TEXT, new PlainLiteralImpl(tag)));
+ //add the user
+ if(user != null){
+ graph.add(new TripleImpl(ta, Properties.DC_CREATOR,user));
+ }
+
+Now the 'graph' contains a valid TextAnnotation for the given user tag. This should be done for all tags of the current content.
+
+In the next step we need to serialize the RDF data. Again I will use here Clerezza as API, but any RDF framework will provide similar functionality
+
+ :::java
+ ByteArrayOutputStream out = new ByteArrayOutputStream();
+ //this tells the Serializer to create "application/rdf+xml"
+ serializer.serialize(out, metadata, SupportedFormat.RDF_XML);
+ String rdfContent = new String(out.toByteArray(),UTF8);
+
+Now we need to create the MultiPart MIME content item containing the metadata and the content
+
+ :::java
+ String content; //the content we want to send to the Stanbol Enhancer
+
+ //the container for the ContentITem
+ MultipartEntity contentItem = new MultipartEntity(null, null ,UTF8);
+
+ //The Metadata MUST BE the first element
+ contentItem.addPart(
+ "metadata", //the name MUST BE "metadata"
+ new StringBody(rdfContent,SupportedFormat.RDF_XML,UTF8){
+ @Override
+ public String getFilename() { //The filename MUST BE the
+ return ciUri.getUnicodeString(); //uri of the ContentItem
+ }
+ });
+
+Note that because the StringBody class provided my the "httpmime" framework does not set a Filename we need to override this method and return the URI of the content item. This is essential, because we need ensure that the URI of the ContentItem is the same as the URI (variable 'ciUri') as used when creating the TextAnnotations for the user tags.
+
+For the following code snippet note that we can directly add the content to the content item container. Only if we would need to sent multiple alternate content versions (as shown in 'Example 3') the usage of an 'multipart/alternate' container is required.
+
+ :::java
+ //Add the content as second mime part
+ contentItem.addPart(
+ "content", //the name MUST BE "content"
+ new StringBody(content,"text/plain",UTF8));
+
+ //now we are ready to create and execute the POST request to the
+ //Stanbol Enhancer
+ HttpPost request = new HttpPost("http://localhost:8080/enhancer");
+ request.setEntity(contentItem);
+ request.setHeader("Accept","application/rdf+xml);
+ Response response = httpClient.execute(request);
+
+The response of the Enhancer will now contain Entity suggestions for the free text user tags.
+