You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@madlib.apache.org by GitBox <gi...@apache.org> on 2019/11/04 22:00:18 UTC
[GitHub] [madlib] fmcquillan99 edited a comment on issue #451: DL: Update
evaluate and predict for multi model outputs
fmcquillan99 edited a comment on issue #451: DL: Update evaluate and predict for multi model outputs
URL: https://github.com/apache/madlib/pull/451#issuecomment-549566751
LGTM
Retesting after last commits (4) (5) (6) (7) (8) (9) (10) below seems OK:
(1) review after recent commit
needed to re-run to get new results:
```
DROP TABLE IF EXISTS iris_multi_model, iris_multi_model_summary, iris_multi_model_info;
SELECT madlib.madlib_keras_fit_multiple_model('iris_train_packed', -- source_table
'iris_multi_model', -- model_output_table
'mst_table', -- model_selection_table
10, -- num_iterations
0, -- use gpus
'iris_test_packed' -- validation dataset
);
SELECT * FROM iris_multi_model_info ORDER BY training_metrics_final DESC;
-[ RECORD 1 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 2
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.16501784324646,0.154597043991089,0.158796072006226,0.171799898147583,0.168801069259644,0.160946846008301,0.168446063995361,0.249520063400269,0.723811149597168,
0.157901048660278}
metrics_type | {accuracy}
training_metrics_final | 0.975000023842
training_loss_final | 0.184436321259
training_metrics | {0.666666686534882,0.708333313465118,0.975000023841858,0.975000023841858,0.458333343267441,0.666666686534882,0.666666686534882,0.716666638851166,0.816666662693024
,0.975000023841858}
training_loss | {0.654977560043335,0.473680555820465,0.407350271940231,0.344571173191071,1.01371514797211,0.536512732505798,0.542378485202789,0.427120715379715,0.301178097724915,
0.184436321258545}
validation_metrics_final | 0.966666638851
validation_loss_final | 0.229953154922
validation_metrics | {0.666666686534882,0.633333325386047,0.933333337306976,0.933333337306976,0.333333343267441,0.666666686534882,0.666666686534882,0.666666686534882,0.800000011920929
,0.966666638851166}
validation_loss | {0.632323026657104,0.544658362865448,0.475189208984375,0.403949052095413,1.15320909023285,0.571010947227478,0.57290381193161,0.472318321466446,0.329961955547333,0
.229953154921532}
-[ RECORD 2 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 6
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.151521921157837,0.170912981033325,0.162657022476196,0.154775142669678,0.152937889099121,0.15539813041687,0.167464017868042,0.155567169189453,0.182074785232544,
0.164788961410522}
metrics_type | {accuracy}
training_metrics_final | 0.933333337307
training_loss_final | 0.653343558311
training_metrics | {0.00833333376795053,0.133333340287209,0.358333319425583,0.491666674613953,0.75,0.958333313465118,0.958333313465118,0.949999988079071,0.908333361148834,0.93333333
7306976}
training_loss | {1.24945497512817,1.10760903358459,1.04750990867615,0.997703850269318,0.913850665092468,0.828234732151031,0.770057380199432,0.728686928749084,0.687141954898834,0.
653343558311462}
validation_metrics_final | 0.833333313465
validation_loss_final | 0.668307900429
validation_metrics | {0,0.100000001490116,0.433333337306976,0.566666662693024,0.766666650772095,0.899999976158142,0.899999976158142,0.866666674613953,0.800000011920929,0.8333333134651
18}
validation_loss | {1.30171895027161,1.11200094223022,1.03035044670105,1.00190925598145,0.922983944416046,0.828430116176605,0.771750569343567,0.736835598945618,0.698546707630157,0.6
68307900428772}
-[ RECORD 3 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 8
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.178833961486816,0.178119897842407,0.193637132644653,0.183725118637085,0.181838989257812,0.190093994140625,0.190335988998413,0.183039903640747,0.189705848693848
,0.196530103683472}
metrics_type | {accuracy}
training_metrics_final | 0.925000011921
training_loss_final | 0.191128313541
training_metrics | {0.675000011920929,0.699999988079071,0.708333313465118,0.850000023841858,0.691666662693024,0.966666638851166,0.949999988079071,0.858333349227905,0.766666650772095
,0.925000011920929}
training_loss | {1.00050187110901,0.477404177188873,0.449770420789719,0.367935240268707,0.715284705162048,0.233321651816368,0.206968143582344,0.247707515954971,0.513520300388336,
0.191128313541412}
validation_metrics_final | 0.899999976158
validation_loss_final | 0.238029867411
validation_metrics | {0.566666662693024,0.666666686534882,0.666666686534882,0.833333313465118,0.566666662693024,0.966666638851166,0.933333337306976,0.866666674613953,0.699999988079071
,0.899999976158142}
validation_loss | {1.10490369796753,0.515014708042145,0.473093926906586,0.429382801055908,0.964671075344086,0.282983720302582,0.234358683228493,0.304766476154327,0.506977200508118,
0.23802986741066}
-[ RECORD 4 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 9
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.179304122924805,0.177167892456055,0.197530031204224,0.186438083648682,0.183886051177979,0.179242849349976,0.261205911636353,0.180387020111084,0.190798997879028
,0.199844121932983}
metrics_type | {accuracy}
training_metrics_final | 0.925000011921
training_loss_final | 0.129962354898
training_metrics | {0.666666686534882,0.875,0.975000023841858,0.758333325386047,0.966666638851166,0.800000011920929,0.966666638851166,0.983333349227905,0.966666638851166,0.925000011
920929}
training_loss | {0.631272912025452,0.311742216348648,0.201621904969215,0.389256983995438,0.119620576500893,0.486419379711151,0.105227634310722,0.0892100557684898,0.13771826028823
9,0.129962354898453}
validation_metrics_final | 1
validation_loss_final | 0.099272236228
validation_metrics | {0.666666686534882,0.833333313465118,0.966666638851166,0.733333349227905,0.899999976158142,0.733333349227905,0.899999976158142,0.966666638851166,0.866666674613953
,1}
validation_loss | {0.655599772930145,0.343398422002792,0.23225425183773,0.460039615631104,0.179262787103653,0.432839661836624,0.164745837450027,0.110677324235439,0.236034005880356,
0.0992722362279892}
-[ RECORD 5 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 12
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.179224967956543,0.178079128265381,0.202226161956787,0.190601825714111,0.179889917373657,0.182010173797607,0.223275899887085,0.186384916305542,0.1926109790802,0
.205245018005371}
metrics_type | {accuracy}
training_metrics_final | 0.916666686535
training_loss_final | 0.557382702827
training_metrics | {0.400000005960464,0.466666668653488,0.641666650772095,0.758333325386047,0.850000023841858,0.875,0.866666674613953,0.908333361148834,0.908333361148834,0.916666686
534882}
training_loss | {1.03677904605865,0.961223006248474,0.898576974868774,0.84269106388092,0.784159600734711,0.72914981842041,0.679587423801422,0.637798070907593,0.592370092868805,0.
557382702827454}
validation_metrics_final | 0.933333337307
validation_loss_final | 0.633718490601
validation_metrics | {0.233333334326744,0.333333343267441,0.433333337306976,0.666666686534882,0.800000011920929,0.933333337306976,1,0.933333337306976,0.966666638851166,0.9333333373069
76}
validation_loss | {1.07957482337952,1.01587510108948,0.959396243095398,0.91132265329361,0.857007622718811,0.808456540107727,0.762722790241241,0.718301713466644,0.671158373355865,0.
633718490600586}
-[ RECORD 6 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 10
model_id | 2
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.185405015945435,0.240031957626343,0.179027080535889,0.188817024230957,0.186041116714478,0.189532041549683,0.191230058670044,0.189625978469849,0.199972867965698
,0.184904098510742}
metrics_type | {accuracy}
training_metrics_final | 0.899999976158
training_loss_final | 0.202067151666
training_metrics | {0.691666662693024,0.675000011920929,0.691666662693024,0.875,0.899999976158142,0.983333349227905,0.983333349227905,0.908333361148834,0.933333337306976,0.899999976
158142}
training_loss | {0.936435461044312,0.705836236476898,0.493957877159119,0.422138810157776,0.360373467206955,0.24227599799633,0.1996029317379,0.195797547698021,0.160537555813789,0.
202067151665688}
validation_metrics_final | 1
validation_loss_final | 0.204844340682
validation_metrics | {0.566666662693024,0.566666662693024,0.566666662693024,0.833333313465118,1,0.966666638851166,0.933333337306976,1,1,1}
validation_loss | {0.968072831630707,0.788067102432251,0.579671025276184,0.491580307483673,0.424855649471283,0.284231871366501,0.230658054351807,0.215303242206573,0.172062397003174
,0.20484434068203}
-[ RECORD 7 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 4
model_id | 1
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=8,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.158287048339844,0.155457973480225,0.576164960861206,0.162032127380371,0.15852689743042,0.156464099884033,0.161139965057373,0.160786867141724,0.167949914932251,
0.656886100769043}
metrics_type | {accuracy}
training_metrics_final | 0.899999976158
training_loss_final | 0.169732123613
training_metrics | {0.666666686534882,0.666666686534882,0.816666662693024,0.991666674613953,0.741666674613953,0.899999976158142,0.925000011920929,0.741666674613953,0.983333349227905
,0.899999976158142}
training_loss | {0.76077276468277,0.5220667719841,0.420222669839859,0.327254116535187,0.341780602931976,0.252596199512482,0.206999227404594,0.435290902853012,0.150588378310204,0.
169732123613358}
validation_metrics_final | 1
validation_loss_final | 0.175337061286
validation_metrics | {0.666666686534882,0.666666686534882,0.666666686534882,0.933333337306976,0.666666686534882,0.833333313465118,1,0.633333325386047,0.966666638851166,1}
validation_loss | {0.767970025539398,0.556378245353699,0.499317765235901,0.392887800931931,0.395217329263687,0.307017415761948,0.249646931886673,0.518399477005005,0.179019212722778
,0.175337061285973}
-[ RECORD 8 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 5
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.152910947799683,0.177506923675537,0.163512945175171,0.153976917266846,0.15569806098938,0.155301809310913,0.166845083236694,0.15528392791748,0.162101984024048,0
.198511123657227}
metrics_type | {accuracy}
training_metrics_final | 0.875
training_loss_final | 0.530299961567
training_metrics | {0.333333343267441,0.191666662693024,0.508333325386047,0.791666686534882,0.883333325386047,0.816666662693024,0.875,0.783333361148834,0.75,0.875}
training_loss | {1.42261958122253,1.11098456382751,0.942022323608398,0.839245975017548,0.778925061225891,0.719573795795441,0.665915071964264,0.617532074451447,0.570941388607025,0
.530299961566925}
validation_metrics_final | 0.866666674614
validation_loss_final | 0.546338915825
validation_metrics | {0.333333343267441,0.16666667163372,0.400000005960464,0.833333313465118,0.899999976158142,0.833333313465118,0.899999976158142,0.833333313465118,0.699999988079071,
0.866666674613953}
validation_loss | {1.58205306529999,1.19787907600403,0.984605967998505,0.874651849269867,0.803973138332367,0.732114672660828,0.681202590465546,0.62757009267807,0.578481078147888,0.
54633891582489}
-[ RECORD 9 ]------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 3
model_id | 1
compile_params | loss='categorical_crossentropy', optimizer='Adam(lr=0.01)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.155992031097412,0.153978109359741,0.572959899902344,0.162055015563965,0.155320167541504,0.156923055648804,0.163105964660645,0.161538124084473,0.168941020965576
,0.654891967773438}
metrics_type | {accuracy}
training_metrics_final | 0.824999988079
training_loss_final | 0.347870081663
training_metrics | {0.633333325386047,0.766666650772095,0.666666686534882,0.666666686534882,0.691666662693024,0.691666662693024,0.908333361148834,0.899999976158142,0.675000011920929
,0.824999988079071}
training_loss | {0.815440833568573,0.520094633102417,0.449681550264359,0.457792103290558,0.437296599149704,0.43956533074379,0.400028377771378,0.407060354948044,0.410939812660217,
0.347870081663132}
validation_metrics_final | 0.800000011921
validation_loss_final | 0.418093115091
validation_metrics | {0.600000023841858,0.666666686534882,0.666666686534882,0.666666686534882,0.566666662693024,0.566666662693024,0.833333313465118,0.866666674613953,0.666666686534882
,0.800000011920929}
validation_loss | {0.782467901706696,0.585534811019897,0.514401435852051,0.50047355890274,0.542211294174194,0.564250826835632,0.486649245023727,0.488315939903259,0.454440474510193,
0.418093115091324}
-[ RECORD 10 ]-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 1
model_id | 1
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 0.7900390625
metrics_elapsed_time | {0.154391050338745,0.155292987823486,0.166854858398438,0.157708883285522,0.15703010559082,0.15372896194458,0.161499977111816,0.15468692779541,0.163357973098755,0.
166243076324463}
metrics_type | {accuracy}
training_metrics_final | 0.716666638851
training_loss_final | 0.498635411263
training_metrics | {0.666666686534882,0.666666686534882,0.816666662693024,0.991666674613953,0.824999988079071,0.933333337306976,0.824999988079071,0.975000023841858,0.816666662693024
,0.716666638851166}
training_loss | {0.597955822944641,0.43520125746727,0.371508300304413,0.0662390738725662,0.332939654588699,0.406542897224426,0.290035516023636,0.127334654331207,0.290014386177063
,0.498635411262512}
validation_metrics_final | 0.699999988079
validation_loss_final | 0.547126591206
validation_metrics | {0.666666686534882,0.666666686534882,0.800000011920929,0.933333337306976,0.800000011920929,1,0.766666650772095,0.933333337306976,0.833333313465118,0.6999999880790
71}
validation_loss | {0.594180822372437,0.464261531829834,0.383160322904587,0.0937399193644524,0.414926111698151,0.368867307901382,0.376850306987762,0.15509657561779,0.317675232887268
,0.547126591205597}
-[ RECORD 11 ]-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 11
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.001)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.189064979553223,0.181553840637207,0.179675817489624,0.196763038635254,0.186076879501343,0.191986083984375,0.194756984710693,0.19181489944458,0.202284812927246,
0.188621997833252}
metrics_type | {accuracy}
training_metrics_final | 0.708333313465
training_loss_final | 0.781156182289
training_metrics | {0.333333343267441,0.333333343267441,0.333333343267441,0.308333337306976,0.333333343267441,0.333333343267441,0.474999994039536,0.758333325386047,0.75,0.7083333134
65118}
training_loss | {1.1848406791687,1.14267539978027,1.11269533634186,1.09739363193512,1.09032690525055,1.08066284656525,1.06443786621094,0.940568923950195,0.859182178974152,0.78115
6182289124}
validation_metrics_final | 0.666666686535
validation_loss_final | 0.827600717545
validation_metrics | {0.333333343267441,0.333333343267441,0.333333343267441,0.300000011920929,0.333333343267441,0.333333343267441,0.566666662693024,0.766666650772095,0.866666674613953
,0.666666686534882}
validation_loss | {1.15087306499481,1.12277579307556,1.10153806209564,1.09295105934143,1.08225655555725,1.07222008705139,1.05354011058807,0.967542886734009,0.901465713977814,0.8276
00717544556}
-[ RECORD 12 ]-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
mst_key | 7
model_id | 2
compile_params | loss='categorical_crossentropy',optimizer='Adam(lr=0.1)',metrics=['accuracy']
fit_params | batch_size=4,epochs=1
model_type | madlib_keras
model_size | 1.2197265625
metrics_elapsed_time | {0.178417921066284,0.60815691947937,0.202095985412598,0.178900003433228,0.18206000328064,0.181649923324585,0.191691875457764,0.182677984237671,0.183932065963745,0
.189748048782349}
metrics_type | {accuracy}
training_metrics_final | 0.358333319426
training_loss_final | 1.10466384888
training_metrics | {0.358333319425583,0.358333319425583,0.358333319425583,0.308333337306976,0.358333319425583,0.358333319425583,0.358333319425583,0.358333319425583,0.333333343267441
,0.358333319425583}
training_loss | {1.10114789009094,1.10259783267975,1.09921455383301,1.1069358587265,1.09716868400574,1.10427033901215,1.0967606306076,1.10034203529358,1.09822237491608,1.10466384
887695}
validation_metrics_final | 0.233333334327
validation_loss_final | 1.14768576622
validation_metrics | {0.233333334326744,0.233333334326744,0.233333334326744,0.433333337306976,0.233333334326744,0.233333334326744,0.233333334326744,0.233333334326744,0.333333343267441
,0.233333334326744}
validation_loss | {1.14770805835724,1.11313581466675,1.09988021850586,1.10149800777435,1.10710287094116,1.10837662220001,1.11643397808075,1.14481270313263,1.10837042331696,1.147685
76622009}
```
(4) review after recent commit
predict on training data
```
DROP TABLE IF EXISTS iris_predict;
SELECT madlib.madlib_keras_predict('iris_multi_model', -- model
'iris_train', -- train_table
'id', -- id column
'attributes', -- independent var
'iris_predict', -- output table
'response', -- prediction type
NULL, -- use gpus
2 -- mst_key to use
);
SELECT COUNT(*) FROM iris_predict JOIN iris_train USING (id)
WHERE iris_predict.estimated_class_text != iris_train.class_text;
-[ RECORD 1 ]
count | 3
```
error %
```
SELECT round(count(*)*100/(150*0.8),2) as train_accuracy_percent from
(select iris_train.class_text as actual, iris_predict.estimated_class_text as estimated
from iris_predict inner join iris_train
on iris_train.id=iris_predict.id) q
WHERE q.actual=q.estimated;
train_accuracy_percent
------------------------
97.50
(1 row)
```
This now matches
```
training_metrics_final | 0.975000023842
```
from the training output table.
OK
(5) review after recent commit
predict on test data
```
DROP TABLE IF EXISTS iris_predict;
SELECT madlib.madlib_keras_predict('iris_multi_model', -- model
'iris_test', -- test_table
'id', -- id column
'attributes', -- independent var
'iris_predict', -- output table
'response', -- prediction type
NULL, -- use gpus
2 -- mst_key to use
);
SELECT COUNT(*) FROM iris_predict JOIN iris_test USING (id)
WHERE iris_predict.estimated_class_text != iris_test.class_text;
count
-------
1
(1 row)
```
as a %
```
SELECT round(count(*)*100/(150*0.2),2) as test_accuracy_percent from
(select iris_test.class_text as actual, iris_predict.estimated_class_text as estimated
from iris_predict inner join iris_test
on iris_test.id=iris_predict.id) q
WHERE q.actual=q.estimated;
test_accuracy_percent
-----------------------
96.67
(1 row)
```
which matches:
```
validation_metrics_final | 0.966666638851
```
from the training output table.
OK
(7) review after last commit
pick wrong model
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model', -- model
'iris_test_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
999 -- mst_key to use
);
ERROR: plpy.Error: madlib_keras_evaluate: mst_key 999 does not exist in the info table (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_evaluate", line 21, in <module>
return madlib_keras.evaluate(**globals())
PL/Python function "madlib_keras_evaluate", line 575, in evaluate
PL/Python function "madlib_keras_evaluate", line 244, in create_summary_view
PL/Python function "madlib_keras_evaluate"
```
which is a better message
OK
(8) review after last commit
input table does not exist
```
DROP TABLE IF EXISTS iris_validate;
SELECT madlib.madlib_keras_evaluate('iris_multi_model999', -- model
'iris_test_packed', -- test table
'iris_validate', -- output table
NULL, -- use gpus
2 -- mst_key to use
);
ERROR: plpy.Error: madlib_keras_evaluate: Missing summary and/or info tables for iris_multi_model999 (plpython.c:5038)
CONTEXT: Traceback (most recent call last):
PL/Python function "madlib_keras_evaluate", line 21, in <module>
return madlib_keras.evaluate(**globals())
PL/Python function "madlib_keras_evaluate", line 575, in evaluate
PL/Python function "madlib_keras_evaluate", line 237, in create_summary_view
PL/Python function "madlib_keras_evaluate"
```
which is a better message
OK
(9) review after last commit:
spot check on errors looks OK
(10) review after last commit:
```
SELECT * FROM iris_multi_model_summary;
-[ RECORD 1 ]-------+---------------------------------------------
source_table | iris_train_packed
validation_table | iris_test_packed
dependent_varname | class_text
independent_varname | attributes
model_arch_table | model_arch_library
num_iterations | 10
start_training_time | 2019-11-04 21:46:55.109008
end_training_time | 2019-11-04 21:49:41.586433
madlib_version | 1.17-dev
num_classes | 3
class_values | {Iris-setosa,Iris-versicolor,Iris-virginica}
dependent_vartype | character varying
normalizing_const | 1
```
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services