Need your opinion to my evaluation plan

[Alexey Grachev <al...@turbinekreuzberg.com>]

Hello dear group.
Kenneth proposed my two ways of solution for creating evaluation for 
complimentary purchase template.

1) create the evaluator based on the recommendation as reference. main 
difference you dont need to filter high rating to define your actual output.

---> To that variant, there is a WIP MR already,  I understand, that I 
cant count on it be reviewed asap, so the second way Kenneth proposed me:

2) write your own evaluation logic separately using other language. . eg 
use pio to run batch prediction, dump it to a file and create another 
file which contain actual item bought together. write a script to read 
both files and generate evaluation metrics

-----> To that variant - I will try to implement ist meanwhile, but 
before I start I would like to ask you, if I understood right, what 
Kenneth proposed me with w):

So:
I created plan for variant 2 (after long research I decided to use 80/20 
cross validation with MAP@OK)

1. shuffle the prepared_json with training data
2. split result of 1 in two files: one with first 80& of data, another 
with last 20% of data and save both.
3. clone complimentary again and call the folder MyComplimentaryPurchaseEval
4. build engine and train the model with the 80%  file.

5. create a file for batch prediction, containing
lines like {"items" : ["s2i1"], "num" : 3}
This batch predict input file has to contain all items, which are in 
"80% file"

6. Run batch prediction for "80% file" (which was used for training) and 
persist result of batch prediction for "80%" file separately
7. Run batch prediction for the "untrained" "20%" file (so for the data 
which was NOT used for the model training) and persist the prediction 
result as well.
8 write php script which compares file from 6. and 7. calculates MAP@OK 
and outputs it.

Please, be so kind and tell me, if my plan for the way 2) is correct.

Thank you in advance,

Alexey