<> AttributeDef string validation

[David Radley <da...@uk.ibm.com>]

Hi all,
I have been having an email conversation with William  (Songqing Ding). 
William is looking for some Atlas work to do. I suggested he looks at 
introducing new AttributeDefs that allow strings to be validated using 
regex patterns; Richard has responded with other helpful considerations. 
This requirement is something that Madhan and others were supportive of in 
a meeting a couple of weeks ago.

The below is the email exchange we have had to date. This email is to 
invite the community to contribute to this design discussion, 
    many thanks,  David. 

----- Forwarded by David Radley/UK/IBM on 08/08/2017 10:12 -----

From:   David Radley/UK/IBM
To:     Songqing Ding/Silicon Valley/IBM@IBMUS
Cc:     Graham Wallis/UK/IBM@IBMGB, Kelvin Lawrence/Austin/IBM@IBMUS, 
Mandy Chessell/UK/IBM@IBMGB, Nigel L Jones/UK/IBM@IBMGB
Date:   08/08/2017 10:12
Subject:        Re: Fw: Atlas Jiras and features


Hi Richard, 

Some comments in <DAVID> tags. 
   all the best David. 




From:   Songqing Ding/Silicon Valley/IBM
To:     David Radley/UK/IBM@IBMGB
Cc:     Graham Wallis/UK/IBM@IBMGB, Kelvin Lawrence/Austin/IBM@IBMUS, 
Mandy Chessell/UK/IBM@IBMGB, Nigel L Jones/UK/IBM@IBMGB
Date:   08/08/2017 00:46
Subject:        Re: Fw: Atlas Jiras and features



Hi David,
 
Thanks for the info, it is very helpful. 
 
I took a look at these Jiras:
 
ATLAS-1955: Validation for Attributes
Each Atlas Type (e.g. String, Date, Struct, ...) currently has a 
isValidValue(Object obj) method and attribute values are validated using 
attribute types. So if Atlas defines a static EMail type, he or she can 
implements isValidValue to validate the email addresses. <DAVID> My 
thinking was that we should make the new attribtutedef validation, data 
driven and declarative and not require programming. If people want very 
complex validation that cannot be easily represented as a pattern - then 
they could look at the programming approach. I think we open up the 
ability to add custom validation - if all they need to do is understand 
regex. Defining the validation in metadata also means that it is managed 
with the other metadata like EntityDefs, whereas managing custom code 
fragments that may not be in the master would become a headache.  </DAVID> 
 
If, say, different organizations have different email formats, then one 
wants to define dynamic email attributes. The existing facility on can use 
is AtlasConstraintDef. Currently each AtlasAttributeDef contains a list of 
AtlasConstraintDefs. However only two types of constraints (ownedRef & 
inverseRef) are defined and constraints are not used to validate the 
attribute values. Therefore, one way to implement a pattern matching 
validation for attributes is to add a PATTERN_MATCH constraint type and 
check the constraint during the value validation process. <DAVID> I 
suggest we have a basic email pattern, which looks for characters followed 
by an @ followed by string segments separated by dots. If an organization 
wants to have a more specialized or different mail format - for example a 
Lotus notes format - they just define a new type with a different pattern. 
 With the proposed approach, we supply a attributedef that 90% could take 
and the other 10% have the flexibility to create their own versions if 
they want to.   </DAVID>
The drawback of the 2nd approach is that the constraints are local to the 
entity types. One needs to define the same constraint in multiple entity 
types if needed. 
ATLAS-1953:
 
It is good idea to have icon attributes so that users can define icons 
associated with their entity types. An new icon type can encapsulate this 
requirement.
 
Thanks,
-Richard
 
 
----- Original message -----
From: David Radley/UK/IBM
To: Kelvin Lawrence/Austin/IBM@IBMUS
Cc: Graham Wallis/UK/IBM@IBMGB, Mandy Chessell/UK/IBM@IBMGB, Songqing 
Ding/Silicon Valley/IBM@IBMUS, Nigel L Jones/UK/IBM@IBMGB
Subject: Fw: Atlas Jiras and features
Date: Mon, Aug 7, 2017 4:05 AM
 
Hi Kelvin,
I am back now. I noticed Richard has been active on the Jiras. I have an 
idea that Richard could look at; in the last call I attended, I mentioned 
some features, that would be great to get in, which the team supported. I 
was thinking Richard could do something along the lines of (draft 
unreviewed design follows .... :-) ) : 

1) Adding an optional icon to EntityDefs
2) To do this is would be good to have a new ValidatableStringDefs  that 
could police the value of the attribute, the idea is that 
ValidatableStringDefs could be CRUD in a similar way to EntityDefs. These 
ValidatableStringDefs would have mandatory Regex expressions and possibly 
a customer validation message. 
3) New ValidatableAttributeDefs would have have specific Regex expressions 
in. Atlas can ship email, url, image AttributeDefs (and others?) . 
4)  The optional icon in EntityDef could then have a type IMAGE. 
5) Due to the inability to validate and reject content during imports - we 
are looking to accept string content - and return warning not error return 
codes indicating that a value was not the expected format. I recall we 
talked of a Kafka message being issued to indicate this has occurred. 
6) This will be useful for the new extended types; as we/he could add 
icons for terms and categories and the like that UI apps could then use to 
should a much nicer UI. 

I think this would be reasonably contained function to bring Richard into 
the project; this work could be split up into a sequence of small 
reviewable tasks. 

Existing Jiras in this space are  ATLAS-1953. and ATLAS-1955. 

If this appeals to Richard - I could send this note copying the dev list - 
so the community can feedback and Madhan is aware - in case HW have / are 
thinking of picking this up. 

all the best, David.  


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 
741598. 
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU