[Devicemap Wiki] Update of "DataSpec2" by rezan

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Devicemap Wiki" for change notification.

The "DataSpec2" page has been changed by rezan:
https://wiki.apache.org/devicemap/DataSpec2?action=diff&rev1=41&rev2=42

  
  
  
+ = Test Suites =
+ 
+ Test suites can be configured to test input against a client on a specific domain.
+ 
+ Test suite attributes:
+ 
+  Domain::
+  :: Type: string
+  :: Required.
+ 
+  DomainVersion::
+  :: Type: string
+  :: Required.
+ 
+  Tests::
+  :: Type: a list of tests
+  :: Required.
+ 
+ Test attributes:
+ 
+  Input::
+  :: Type: string
+  :: Required.
+ 
+  ResultPatternId::
+  :: Type: string
+  :: Required.
+ 
+  ResultAttributes::
+  :: Type: a list of attributes and values
+  :: Optional.
+ 
+ When a client runs a test suite, it needs to load the associated domain and then iterate thru each test,
+ running the input string against the domain and then checking the resulting PatternId and attributes against
+ the expected results.
+ 
+ A client must report several metrics on completion of the test suite:
+ 
+  * Number of tests attempted
+  * Number of tests failed
+  * Total runtime in ms, including domain loadtime
+  * [Optional] Amount of memory required to run the test suite in bytes
+ 
+ A client passes a test suite if there are no failed tests.
+ 
+ A reference domain and test suite will be created along with a reference client.
+ All clients should pass the reference test suite to be considered a valid client.
+ All domains should provide a rigorous test suite which passes on the reference client to be considered a valid domain.
+ Any valid clients can run any valid domain.
+ 
+ 
+ 
  = Format =
  
  The pattern and attribute files are JSON objects. These files will contain: