Technical Debt

[Benjamin Mahler <bm...@apache.org>]

Hi devs,

TLDR: I created a document with the help of other committers and
contributors to collect technical debt items, categorized by the benefit we
get if we fix them. If you'd like to contribute to any of these areas,
please reach out! Also feel free to make suggestions / comments on the doc:

https://docs.google.com/document/d/103jyXD06jdq_puGFmPQUXjRuqj_b-
lGLmRn6wqx9ejc/edit?usp=sharing

---

As we grow the project, "technical debt" [1] naturally accumulates.
Wikipedia defines technical debt as: "a concept in programming that
reflects the extra development work that arises when code that is easy to
implement in the short run is used instead of applying the best overall
solution".

In terms of having useful tracking, I think we should use the term
"technical debt" more broadly to include anything that slows down
development, diminishes the user experience, diminishes the quality of the
system, regardless of whether it arose from applying a short term solution.

You could categorize our approaches to dealing with technical debt into two
strategies:

(1) Pay down debt as part of feature development. This approach allows us
to better sustain development by trading off short term development
throughput/latency for long term development throughput.

(2) Pay down debt as an independent effort. This approach sacrifices short
term development throughput but can avoid having an effect on short term
development latency.

I think we need to use both techniques in order to sustain the project. (1)
often doesn't happen because it slows down feature development, and (2)
often doesn't happen because people move on to other features.

As an effort to do more of (2), I started a technical debt doc with the
help of other committers and contributors document to track things we'd
like to improve, categorized by the benefit we get if we fix them:

https://docs.google.com/document/d/103jyXD06jdq_puGFmPQUXjRuqj_b-
lGLmRn6wqx9ejc/edit?usp=sharing

I still haven't linked/created tickets for the items, and ideally we don't
need this on top of JIRA, but I think it's a useful organization for now.
If you'd like to contribute to any of these please reach out!

Ben

[1] https://en.wikipedia.org/wiki/Technical_debt