Notes from the contributor meeting

[Olga Natkovich <ol...@yahoo-inc.com>]

Hi,

Attendees:

Dmitriy Ryaboy
Alan Gates
Ashutosh Chauhan
Daniel Dai
Xuefu Zhang
Richard Ding
Olga Natkovich

Topics discussed:


(1)    Improving Pig testing:

a.       Short term

                                                               i.      making tests run significantly faster. Dmitriy said he would work on transitioning the tests into local mode. Hopefully that will reduce the run time from 10 hours to about 3.

                                                             ii.      Get test patch automation back on. I took an action item to follow up on this.

b.      Longer term

                                                               i.      Move beyond unit testing. Alan suggested that's once recently open sourced e2e harness is ready to be used (3-6 month) we would move most of e2e tests we currently run as unit tests into the e2e tests and only leave true unit tests in JUnit. This will reduce unit test runtime to something under an hour and will allow to run the e2e tests on real data and real clusters making the testing more realistic.

                                                             ii.      Figuring out a way to make UDF testing easier. I don't think we had many good ideas on how to do this. Needs further discussion

(2)    Discussion on release management. Main goal is to maintain stability for production systems while allowing changes to be released quickly. We came up with the following proposal:

a.       Making major releases time (not feature) based and release every 3 month

b.      Make sure that branches post release are kept stable by only allowing P1 changes (failures with no reasonable workaround or silent failures)

c.       Develop disruptive features (example - parser changes) on separate branches and only folding them in once the code was completed and stabilized.

(3)    Discussion on revamping UDF interface

a.       Making interface simpler - no need to implement 3 different version

b.      Making it more intuitive

                                                               i.      No need for wrapping input parameters into tuples

                                                             ii.      No need for parameters casting

                                                            iii.      Simplify schema management

                                                           iv.      Simplify overloading

c.       This will need to coexist with the current approach for a significant amount of time (6-12 month) to let users transition.

(4)    Status of Piggybank

a.       Not much progress so far. Dmitriy is struggling with the build process.


Other attendees - please, feel free to add.

Olga