[GitHub] [flink-benchmarks] fredia commented on pull request #66: [FLINK-29825] Replace median-based regression detect algorithm with max/min-based

["fredia (via GitHub)" <gi...@apache.org>]

fredia commented on PR #66:
URL: https://github.com/apache/flink-benchmarks/pull/66#issuecomment-1463387085

   @lindong28  Thanks for the review and suggestions.
   
   > do you have numbers showing how this algorithm performs in production? For example, is there false positive and false negative rate of this algorithm?
   
   I have selected several benchmarks to test the precision and recall of this algorithm. Here are some data:
   
   <!--text_ne_inode_%7B%22type%22%3A%22fragment%22%2C%22name%22%3A%22%23fragment%22%2C%22children%22%3A%5B%7B%22id%22%3A%22%22%2C%22type%22%3A%22element%22%2C%22name%22%3A%22table%22%2C%22attrs%22%3A%7B%22rowCount%22%3A4%2C%22colCount%22%3A7%2C%22colWidths%22%3A%5B212%2C120%2C121%2C125%2C118%2C93%2C93%5D%7D%2C%22children%22%3A%5B%7B%22id%22%3A%22%22%2C%22type%22%3A%22element%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uede0c1ce%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%7D%2C%22children%22%3A%5B%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u26637314%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ud8922fe7%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22uc8a2b1eb%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22tru
 e%20positive%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u27ad74aa%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u0db61f98%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u9480247e%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22false%20positive%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u0692fb83%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A3%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uaaa9c5a7%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u87e063e9%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22true%20negative%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u444faacc%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%
 22col%22%3A4%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uaac84697%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u7c5c2750%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22false%20negative%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubc0b4b93%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A5%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u5ddd9d93%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u5375c62e%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22precision%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u2c7e3999%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A6%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u6603027f%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D
 %2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ufa25865b%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22recall%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22id%22%3A%22%22%2C%22type%22%3A%22element%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubf27f27a%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u9cce2406%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u594b94d9%22%2C%22name%22%3A%22link%22%2C%22attrs%22%3A%7B%22src%22%3A%22http%3A%2F%2Fcodespeed.dak8s.net%3A8000%2Ftimeline%2F%23%2F%3Fexe%3D1%2C5%2C8%2C9%26ben%3DarrayKeyBy%26env%3D2%26revs%3D1000%26equid%3Doff%26quarts%3Don%26extr%3Don%22%2C%22external%22%3Atrue%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u136d9871%22%2C%22name%22%3A%22%23text%22%2C%22at
 trs%22%3A%7B%7D%2C%22data%22%3A%22arrayKeyBy%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ufe7a793a%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u85359969%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua196047c%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%2242%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u20cb2aa0%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u6769bc89%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u666234e5%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%222%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uf49a1366%22%2C%22name%22%3A%22td%22%2C%22attrs
 %22%3A%7B%22col%22%3A3%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u1f8626fb%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u219a5c52%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22840%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u764e4d5b%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A4%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ua9d5b121%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u11c3654d%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%2210%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uff130b4c%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A5%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ufc61aff7%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22chil
 dren%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua11d2434%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.96%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ubd21aa3b%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A6%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u04634a48%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u7513a3f5%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.83%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22id%22%3A%22%22%2C%22type%22%3A%22element%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uacc5d8a6%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u6f24e585%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22el
 ement%22%2C%22id%22%3A%22uee355f3c%22%2C%22name%22%3A%22link%22%2C%22attrs%22%3A%7B%22src%22%3A%22http%3A%2F%2Fcodespeed.dak8s.net%3A8000%2Ftimeline%2F%23%2F%3Fexe%3D5%2C6%2C8%2C9%26ben%3DserializerRow%26env%3D2%26revs%3D1000%26equid%3Doff%26quarts%3Don%26extr%3Don%22%2C%22external%22%3Atrue%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ue817317b%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22serializerRow_Java11%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u320592b7%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A1%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u946b5fd0%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua498dc74%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%224%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u3484eca9%22%2C%22name%22%3A%22td%22%2C%22at
 trs%22%3A%7B%22col%22%3A2%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u77e746b9%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u86498762%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%221%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u047fa43b%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A3%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uf40a40a3%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u0043d658%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22241%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ue61385ce%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A4%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u093dde1b%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22ch
 ildren%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ua3faa3bc%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%223%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u34067aa9%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A5%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u8a64ac66%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22uaa98ec3d%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.8%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u753c9ee7%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A6%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u28de1245%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u213be29e%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.57%22%7D%5D%7D%5D%7D
 %5D%7D%2C%7B%22id%22%3A%22%22%2C%22type%22%3A%22element%22%2C%22name%22%3A%22tr%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ue9dd73bf%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A0%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ud417f598%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uf8705413%22%2C%22name%22%3A%22link%22%2C%22attrs%22%3A%7B%22src%22%3A%22http%3A%2F%2Fcodespeed.dak8s.net%3A8000%2Ftimeline%2F%23%2F%3Fexe%3D5%2C6%2C8%2C9%26ben%3DserializerTuple%26env%3D2%26revs%3D1000%26equid%3Doff%26quarts%3Don%26extr%3Don%22%2C%22external%22%3Atrue%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u206d83a4%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22serializerTuple_Java11%22%7D%5D%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u4adcdb59%22%2C%22name%22%3A%22td%22%2C
 %22attrs%22%3A%7B%22col%22%3A1%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ue598a6d6%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u48c7bea9%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%224%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u52e313fe%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A2%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ud90aed0b%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ud1f15dfc%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%2224%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u89b83737%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A3%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uc9c5883d%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%
 22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u0c4e7e22%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%22218%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22ufdf705ef%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A4%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u182a9483%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22uc8645081%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%223%22%7D%5D%7D%5D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u22c69650%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A5%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uacf6b02b%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22ufbf77fd0%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.14%22%7D%5D%7D%5
 D%7D%2C%7B%22type%22%3A%22element%22%2C%22id%22%3A%22u77c61abc%22%2C%22name%22%3A%22td%22%2C%22attrs%22%3A%7B%22col%22%3A6%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22element%22%2C%22id%22%3A%22uee43a543%22%2C%22name%22%3A%22p%22%2C%22attrs%22%3A%7B%7D%2C%22children%22%3A%5B%7B%22type%22%3A%22text%22%2C%22id%22%3A%22u3c387c19%22%2C%22name%22%3A%22%23text%22%2C%22attrs%22%3A%7B%7D%2C%22data%22%3A%220.57%22%7D%5D%7D%5D%7D%5D%7D%5D%7D%5D%2C%22attrs%22%3A%7B%7D%7D--><div class="lake-content" typography="classic">
   
     | true positive | false positive | true negative | false negative | precision | recall
   -- | -- | -- | -- | -- | -- | --
   [arrayKeyBy](http://codespeed.dak8s.net:8000/timeline/#/?exe=1,5,8,9&ben=arrayKeyBy&env=2&revs=1000&equid=off&quarts=on&extr=on) | 42 | 2 | 840 | 10 | 0.96 | 0.83
   [serializerRow_Java11](http://codespeed.dak8s.net:8000/timeline/#/?exe=5,6,8,9&ben=serializerRow&env=2&revs=1000&equid=off&quarts=on&extr=on) | 4 | 1 | 241 | 3 | 0.8 | 0.57
   [serializerTuple_Java11](http://codespeed.dak8s.net:8000/timeline/#/?exe=5,6,8,9&ben=serializerTuple&env=2&revs=1000&equid=off&quarts=on&extr=on) | 4 | 24 | 218 | 3 | 0.14 | 0.57
   
   Overall, It performs differently on different benchmarks, but its precision and recall can basically be greater than 50%.
   
   Here is a [doc](https://docs.google.com/document/d/1coI4eJsauBtrlS1Z77bhGf-hNtDEXbzuwacG5ZPCMc8/edit?usp=sharing) comparing it to the current median-based algorithm, which contains their false positive and false negative data on different types of benchmarks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org