You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flagon.apache.org by Evan Jones <ev...@gmail.com> on 2024/01/17 00:36:41 UTC
[RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Hi all,
I've recently had more bandwidth to contribute to Flagon. Recently, I
helped with the release of Flagon-Distill v0.1.0 and am leading up the
instrumentation team at ARLIS. My goal is for us to ramp up our
contribution to the Flagon suite.
After spending quite a bit of time thinking about Flagon, I've come to
believe the best way for Flagon to deliver truly differentiated value to
end users is to deliver on the original vision of all the sub-projects:
- UserALE for instrumentation
- Distill for analytics
- STOUT as a UI for the analytics, metrics, and experiment
- TAP (re-envisioned) as the infrastructure platform to connect all
those pieces
By eventually building all these pieces, Flagon could be a very strong
open-source alternative to services like Google Analytics, LogRocket, and
others. I intend to socialize the above vision more in the future. For now
I digress since that's not the point of this email.
The point of this email is to share an RFC
<https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing>
for *an overhaul I believe needs to happen as a precursor to achieving the
above vision*.
*Please share feedback, comments and concerns right on the google doc*.
I've also opened a GH issue <https://github.com/apache/flagon/issues/44> if
folks would like to contribute to the conversation there, though I prefer
to keep as much commentary within the document itself to simplify collation
later.
Many thanks!
Best
Evan Jones
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Evan Jones <ev...@gmail.com>.
Josh -- Thanks for looking into the ASF process side. This was the biggest
unknown to me.
Also,
> I’m pretty sure we can generate Pypi and NPM releases from the same repo,
> different folders.
100% doable. Apache Beam is a good example of this. They use gradle as a
global build manager for the language specific SDKs. Bazel, as mentioned in
the RFC, is another option. However, both of those are probably overkill
for us at this point. We could probably get by with just using Makefiles to
standardize build and test across all the projects.
Best
Evan Jones
Website: www.ea-jones.com
On Tue, Jan 30, 2024 at 5:18 PM Joshua Poore <po...@apache.org> wrote:
> The idea is growing on me—I’m pretty sure we can generate Pypi and NPM
> releases from the same repo, different folders.
>
> Agree that now is the time to do it before user base on Distill grows.
>
> I do think we should run a consensus/community VOTE on this. Let me
> refresh myself on which ruleset we should use for this. It’s a big change,
> so may need to fall under the same guidelines as RELEASE.
>
> Josh
>
> > On Jan 30, 2024, at 4:05 PM, Evan Jones <ev...@gmail.com> wrote:
> >
> > *When should this process begin? *In terms of where UserALE and currently
> > are.**
> > I want to get all *non-breaking* PRs merged into UserALE/test and
> > Distill/main and then do a new release of each of them, assuming there
> are
> > changes to release. UserALE has been queued up for a 2.4.0 for quite some
> > time. I don't think we have anything new for Distill. That way the
> monorepo
> > consolidation would happen on top of clean releases.
> >
> > I want to start the monorepo migration ASAP which means getting these
> > releases in progress ASAP.
> >
> > *The scope and sequence of the changes? Roadmap for the migration?*
> > If by scope you mean what aspects of the code base does this touch and
> > who's impacted, then the answer to the former is all aspects -- albeit
> > purely in a structural way -- and the answer to the latter is
> > contributors/devs only. Nothing about our individual projects will change
> > to start except they'll all be moved into the Flagon repo as
> > sub-directories. Within those directories, everything will be identical
> to
> > how they are currently. The only real repo that will see changes is the
> > Flagon repo.
> >
> > Because of this consolidation, the way developers contribute to the
> > projects will change. People will fork the Flagon Suite repo and merge
> back
> > into it regardless of the flagon product (distill, userale) they're
> > contributing to. Each project will still be released the same way.
> >
> > On the ASF side, we would have a single release process for the entire
> > Flagon project, rather than different releases for different
> sub-projects.
> > My hope is this vastly lowers overhead.
> >
> > The steps proposed above are the roadmap, more or less. As far as
> timeline
> > goes, I think the biggest bottleneck will be getting the current releases
> > out. I believe I can do the consolidation into a single repo in one
> weekend.
> >
> > *For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> others
> > that you might know of?*
> >
> > All documentation and website files would be stored in a single directory
> > in the repo. It sits alongside userale, distill, etc. The API
> documentation
> > for each project should leverage automated doc generators:
> >
> > - Distill = Sphinx + PyDoc --> ReadTheDocs
> > - UserALE = Sphinx + JsDoc --> ReadTheDocs
> >
> > The main flagon website should be the primary source of documentation for
> > things like tutorials, getting started, concept definitions, etc. It can
> > then point to the ReadTheDocs pages for each of packages' API documents.
> >
> > The overhauling of the documentation is not really couple to the monorepo
> > migration, though. The monorepo migration merely lowers to burden of the
> > workflow I describe above by centralizing everything.
> >
> > *I think we should set up a robust CI/CD pipeline to handle the monorepo
> to
> > ensure that all projects are integrated seamlessly and that automated
> tests
> > are in place for the monorepo.*
> >
> > Yes.
> >
> > Best
> >
> > Evan Jones
> > Website: www.ea-jones.com
> >
> >
> > On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <ag...@umd.edu> wrote:
> >
> >> Hi Evan,
> >>
> >> I support this monorepo structure - as you stated, there are
> >> several benefits of consolidating the Apache Flagon projects. Just a few
> >> questions/suggestions:
> >>
> >> - Do you have a timeline in mind for:
> >>
> >>
> >> - When should this process begin? *In terms of where UserALE and
> Distill
> >> currently are.*
> >> - The scope and sequence of the changes? Roadmap for the migration?
> >>
> >>
> >> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> >> others that you might know of?
> >> - I think we should set up a robust CI/CD pipeline to handle the
> >> monorepo to ensure that all projects are integrated seamlessly and
> that
> >> automated tests are in place for the monorepo.
> >>
> >>
> >> Best,
> >> *Amir M. Ghaemi*
> >>
> >> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com>
> >> wrote:
> >>
> >>> There are likely a few more steps:
> >>>
> >>> 1. Send a do not merge email for all repos
> >>> 2. Add userale/distill code, git tree, and github actions to the flagon
> >>> repo
> >>> 3. *Consolidate documents into a single repo*
> >>> 4. *Migrate to a build tool for monorepos like Bazel* (example:
> >>>
> https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
> >>> Beam uses gradle)
> >>> 5. Mark userale/distill repos as archived and update read me with a
> link
> >>> 6. Send a merge freely email
> >>> 7. Update flagon readme and flagon site links
> >>>
> >>> The bolded things could happen after the structural changes depending
> on
> >>> our priorities. If we implement those changes as part of the migration
> it
> >>> would simplify contribution conventions post migration but slow things
> >> down
> >>> in the short-term. If we implement them after, we'll open back up to
> >> merges
> >>> faster but the repo will be in a "dirty" state for longer as we
> >> transition
> >>> to monorepo best practices.
> >>>
> >>>> I don't think there's anything besides the flagon site that points to
> >> any
> >>> github repo, but I'm not sure.
> >>>
> >>> We definitely want to check our p's and q's though, especially on the
> >>> Apache side. Are the Distill, UserALE and Flagon repos treated as
> >> separate
> >>> projects / releases in the eyes of Apache or are their releases all
> >> bundled
> >>> into Flagon releases?
> >>>
> >>> Best
> >>>
> >>> Evan Jones
> >>> Website: www.ea-jones.com
> >>>
> >>>
> >>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
> >>>
> >>>> I'm all for a monorepo. Especially because it's hard to get visibility
> >>>> into a single project when its spread across multiple repos in the sea
> >> of
> >>>> the Apache GitHub org.
> >>>>
> >>>> Are these what the steps look like for moving to a monorepo?
> >>>>
> >>>> 1. Send a do not merge email for all repos
> >>>> 2. Add userale/distill code, git tree, and github actions to the
> flagon
> >>>> repo
> >>>> 3. Mark userale/distill repos as archived and update read me with a
> >> link
> >>>> 4. Send a merge freely email
> >>>> 5. Update flagon readme and flagon site links
> >>>>
> >>>> I don't think there's anything besides the flagon site that points to
> >> any
> >>>> github repo, but I'm not sure.
> >>>>
> >>>> On 2024/01/17 00:36:41 Evan Jones wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I've recently had more bandwidth to contribute to Flagon. Recently, I
> >>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up
> >> the
> >>>>> instrumentation team at ARLIS. My goal is for us to ramp up our
> >>>>> contribution to the Flagon suite.
> >>>>>
> >>>>> After spending quite a bit of time thinking about Flagon, I've come
> >> to
> >>>>> believe the best way for Flagon to deliver truly differentiated value
> >>> to
> >>>>> end users is to deliver on the original vision of all the
> >> sub-projects:
> >>>>>
> >>>>> - UserALE for instrumentation
> >>>>> - Distill for analytics
> >>>>> - STOUT as a UI for the analytics, metrics, and experiment
> >>>>> - TAP (re-envisioned) as the infrastructure platform to connect
> >> all
> >>>>> those pieces
> >>>>>
> >>>>> By eventually building all these pieces, Flagon could be a very
> >> strong
> >>>>> open-source alternative to services like Google Analytics, LogRocket,
> >>> and
> >>>>> others. I intend to socialize the above vision more in the future.
> >> For
> >>>> now
> >>>>> I digress since that's not the point of this email.
> >>>>>
> >>>>> The point of this email is to share an RFC
> >>>>> <
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> >>>>>
> >>>>> for *an overhaul I believe needs to happen as a precursor to
> >> achieving
> >>>> the
> >>>>> above vision*.
> >>>>>
> >>>>> *Please share feedback, comments and concerns right on the google
> >> doc*.
> >>>>> I've also opened a GH issue <
> >>> https://github.com/apache/flagon/issues/44>
> >>>> if
> >>>>> folks would like to contribute to the conversation there, though I
> >>> prefer
> >>>>> to keep as much commentary within the document itself to simplify
> >>>> collation
> >>>>> later.
> >>>>>
> >>>>> Many thanks!
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> Evan Jones
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Evan Jones <ev...@gmail.com>.
>
> has anyone done the
> equivalent planning work to actually surface a pros/cons; ex: what would
> the tooling need to look like to address the problems and maintain the
> current repo structure?
Not that I know of, but it's worth exploring first.
I am not sure I understand how documentation improves by having a monorepo.
Documentation doesn't magically improve. You still have to do the work.
it seems just as straightforward to invest in the ability to "treat
> a number of highly inter-related projects as" coupled,
Guidance on this more straightforward way? This RFC contends a monorepo is
the straight-forward ability.
Best
Evan Jones
Website: www.ea-jones.com
On Tue, Jan 30, 2024 at 8:16 PM Austin Bennett <au...@apache.org> wrote:
> On: ASF Admin's don't understand project health due to repo structure --
> i disagree with the idea that itshould at all inform/dictate how work gets
> structured and completed by the Flagon project.
>
> I am not sure I understand how documentation improves by having a
> monorepo. I guess if pitched about a single README section. Open Source
> projects commonly have docs that get autogenerated and published to a
> webpage, and can be done so from a number of disparate repos to a single
> location.
>
> Off handed, it seems just as straightforward to invest in the ability to
> "treat
> a number of highly inter-related projects as" coupled, than to refactor
> repos and invest in alternative testing infra. Either way, the problem
> outlined appears actually to surface the urgency for increasingly robust
> CI?
>
> The rest is docs and education, which can be handled in a number of ways.
>
> I don't buy the necessity of a monorepo, but *I accept the preferences of
> those that will choose to do the work*. Wondering: has anyone done the
> equivalent planning work to actually surface a pros/cons; ex: what would
> the tooling need to look like to address the problems and maintain the
> current repo structure?
>
> On Tue, Jan 30, 2024 at 2:18 PM Joshua Poore <po...@apache.org> wrote:
>
> > The idea is growing on me—I’m pretty sure we can generate Pypi and NPM
> > releases from the same repo, different folders.
> >
> > Agree that now is the time to do it before user base on Distill grows.
> >
> > I do think we should run a consensus/community VOTE on this. Let me
> > refresh myself on which ruleset we should use for this. It’s a big
> change,
> > so may need to fall under the same guidelines as RELEASE.
> >
> > Josh
> >
> > > On Jan 30, 2024, at 4:05 PM, Evan Jones <ev...@gmail.com>
> wrote:
> > >
> > > *When should this process begin? *In terms of where UserALE and
> currently
> > > are.**
> > > I want to get all *non-breaking* PRs merged into UserALE/test and
> > > Distill/main and then do a new release of each of them, assuming there
> > are
> > > changes to release. UserALE has been queued up for a 2.4.0 for quite
> some
> > > time. I don't think we have anything new for Distill. That way the
> > monorepo
> > > consolidation would happen on top of clean releases.
> > >
> > > I want to start the monorepo migration ASAP which means getting these
> > > releases in progress ASAP.
> > >
> > > *The scope and sequence of the changes? Roadmap for the migration?*
> > > If by scope you mean what aspects of the code base does this touch and
> > > who's impacted, then the answer to the former is all aspects -- albeit
> > > purely in a structural way -- and the answer to the latter is
> > > contributors/devs only. Nothing about our individual projects will
> change
> > > to start except they'll all be moved into the Flagon repo as
> > > sub-directories. Within those directories, everything will be identical
> > to
> > > how they are currently. The only real repo that will see changes is the
> > > Flagon repo.
> > >
> > > Because of this consolidation, the way developers contribute to the
> > > projects will change. People will fork the Flagon Suite repo and merge
> > back
> > > into it regardless of the flagon product (distill, userale) they're
> > > contributing to. Each project will still be released the same way.
> > >
> > > On the ASF side, we would have a single release process for the entire
> > > Flagon project, rather than different releases for different
> > sub-projects.
> > > My hope is this vastly lowers overhead.
> > >
> > > The steps proposed above are the roadmap, more or less. As far as
> > timeline
> > > goes, I think the biggest bottleneck will be getting the current
> releases
> > > out. I believe I can do the consolidation into a single repo in one
> > weekend.
> > >
> > > *For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> > others
> > > that you might know of?*
> > >
> > > All documentation and website files would be stored in a single
> directory
> > > in the repo. It sits alongside userale, distill, etc. The API
> > documentation
> > > for each project should leverage automated doc generators:
> > >
> > > - Distill = Sphinx + PyDoc --> ReadTheDocs
> > > - UserALE = Sphinx + JsDoc --> ReadTheDocs
> > >
> > > The main flagon website should be the primary source of documentation
> for
> > > things like tutorials, getting started, concept definitions, etc. It
> can
> > > then point to the ReadTheDocs pages for each of packages' API
> documents.
> > >
> > > The overhauling of the documentation is not really couple to the
> monorepo
> > > migration, though. The monorepo migration merely lowers to burden of
> the
> > > workflow I describe above by centralizing everything.
> > >
> > > *I think we should set up a robust CI/CD pipeline to handle the
> monorepo
> > to
> > > ensure that all projects are integrated seamlessly and that automated
> > tests
> > > are in place for the monorepo.*
> > >
> > > Yes.
> > >
> > > Best
> > >
> > > Evan Jones
> > > Website: www.ea-jones.com
> > >
> > >
> > > On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <ag...@umd.edu> wrote:
> > >
> > >> Hi Evan,
> > >>
> > >> I support this monorepo structure - as you stated, there are
> > >> several benefits of consolidating the Apache Flagon projects. Just a
> few
> > >> questions/suggestions:
> > >>
> > >> - Do you have a timeline in mind for:
> > >>
> > >>
> > >> - When should this process begin? *In terms of where UserALE and
> > Distill
> > >> currently are.*
> > >> - The scope and sequence of the changes? Roadmap for the
> migration?
> > >>
> > >>
> > >> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> > >> others that you might know of?
> > >> - I think we should set up a robust CI/CD pipeline to handle the
> > >> monorepo to ensure that all projects are integrated seamlessly and
> > that
> > >> automated tests are in place for the monorepo.
> > >>
> > >>
> > >> Best,
> > >> *Amir M. Ghaemi*
> > >>
> > >> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com>
> > >> wrote:
> > >>
> > >>> There are likely a few more steps:
> > >>>
> > >>> 1. Send a do not merge email for all repos
> > >>> 2. Add userale/distill code, git tree, and github actions to the
> flagon
> > >>> repo
> > >>> 3. *Consolidate documents into a single repo*
> > >>> 4. *Migrate to a build tool for monorepos like Bazel* (example:
> > >>>
> > https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
> > >>> Beam uses gradle)
> > >>> 5. Mark userale/distill repos as archived and update read me with a
> > link
> > >>> 6. Send a merge freely email
> > >>> 7. Update flagon readme and flagon site links
> > >>>
> > >>> The bolded things could happen after the structural changes depending
> > on
> > >>> our priorities. If we implement those changes as part of the
> migration
> > it
> > >>> would simplify contribution conventions post migration but slow
> things
> > >> down
> > >>> in the short-term. If we implement them after, we'll open back up to
> > >> merges
> > >>> faster but the repo will be in a "dirty" state for longer as we
> > >> transition
> > >>> to monorepo best practices.
> > >>>
> > >>>> I don't think there's anything besides the flagon site that points
> to
> > >> any
> > >>> github repo, but I'm not sure.
> > >>>
> > >>> We definitely want to check our p's and q's though, especially on the
> > >>> Apache side. Are the Distill, UserALE and Flagon repos treated as
> > >> separate
> > >>> projects / releases in the eyes of Apache or are their releases all
> > >> bundled
> > >>> into Flagon releases?
> > >>>
> > >>> Best
> > >>>
> > >>> Evan Jones
> > >>> Website: www.ea-jones.com
> > >>>
> > >>>
> > >>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
> > >>>
> > >>>> I'm all for a monorepo. Especially because it's hard to get
> visibility
> > >>>> into a single project when its spread across multiple repos in the
> sea
> > >> of
> > >>>> the Apache GitHub org.
> > >>>>
> > >>>> Are these what the steps look like for moving to a monorepo?
> > >>>>
> > >>>> 1. Send a do not merge email for all repos
> > >>>> 2. Add userale/distill code, git tree, and github actions to the
> > flagon
> > >>>> repo
> > >>>> 3. Mark userale/distill repos as archived and update read me with a
> > >> link
> > >>>> 4. Send a merge freely email
> > >>>> 5. Update flagon readme and flagon site links
> > >>>>
> > >>>> I don't think there's anything besides the flagon site that points
> to
> > >> any
> > >>>> github repo, but I'm not sure.
> > >>>>
> > >>>> On 2024/01/17 00:36:41 Evan Jones wrote:
> > >>>>> Hi all,
> > >>>>>
> > >>>>> I've recently had more bandwidth to contribute to Flagon.
> Recently, I
> > >>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up
> > >> the
> > >>>>> instrumentation team at ARLIS. My goal is for us to ramp up our
> > >>>>> contribution to the Flagon suite.
> > >>>>>
> > >>>>> After spending quite a bit of time thinking about Flagon, I've come
> > >> to
> > >>>>> believe the best way for Flagon to deliver truly differentiated
> value
> > >>> to
> > >>>>> end users is to deliver on the original vision of all the
> > >> sub-projects:
> > >>>>>
> > >>>>> - UserALE for instrumentation
> > >>>>> - Distill for analytics
> > >>>>> - STOUT as a UI for the analytics, metrics, and experiment
> > >>>>> - TAP (re-envisioned) as the infrastructure platform to connect
> > >> all
> > >>>>> those pieces
> > >>>>>
> > >>>>> By eventually building all these pieces, Flagon could be a very
> > >> strong
> > >>>>> open-source alternative to services like Google Analytics,
> LogRocket,
> > >>> and
> > >>>>> others. I intend to socialize the above vision more in the future.
> > >> For
> > >>>> now
> > >>>>> I digress since that's not the point of this email.
> > >>>>>
> > >>>>> The point of this email is to share an RFC
> > >>>>> <
> > >>>>
> > >>>
> > >>
> >
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> > >>>>>
> > >>>>> for *an overhaul I believe needs to happen as a precursor to
> > >> achieving
> > >>>> the
> > >>>>> above vision*.
> > >>>>>
> > >>>>> *Please share feedback, comments and concerns right on the google
> > >> doc*.
> > >>>>> I've also opened a GH issue <
> > >>> https://github.com/apache/flagon/issues/44>
> > >>>> if
> > >>>>> folks would like to contribute to the conversation there, though I
> > >>> prefer
> > >>>>> to keep as much commentary within the document itself to simplify
> > >>>> collation
> > >>>>> later.
> > >>>>>
> > >>>>> Many thanks!
> > >>>>>
> > >>>>> Best
> > >>>>>
> > >>>>> Evan Jones
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Austin Bennett <au...@apache.org>.
On: ASF Admin's don't understand project health due to repo structure --
i disagree with the idea that itshould at all inform/dictate how work gets
structured and completed by the Flagon project.
I am not sure I understand how documentation improves by having a
monorepo. I guess if pitched about a single README section. Open Source
projects commonly have docs that get autogenerated and published to a
webpage, and can be done so from a number of disparate repos to a single
location.
Off handed, it seems just as straightforward to invest in the ability to "treat
a number of highly inter-related projects as" coupled, than to refactor
repos and invest in alternative testing infra. Either way, the problem
outlined appears actually to surface the urgency for increasingly robust
CI?
The rest is docs and education, which can be handled in a number of ways.
I don't buy the necessity of a monorepo, but *I accept the preferences of
those that will choose to do the work*. Wondering: has anyone done the
equivalent planning work to actually surface a pros/cons; ex: what would
the tooling need to look like to address the problems and maintain the
current repo structure?
On Tue, Jan 30, 2024 at 2:18 PM Joshua Poore <po...@apache.org> wrote:
> The idea is growing on me—I’m pretty sure we can generate Pypi and NPM
> releases from the same repo, different folders.
>
> Agree that now is the time to do it before user base on Distill grows.
>
> I do think we should run a consensus/community VOTE on this. Let me
> refresh myself on which ruleset we should use for this. It’s a big change,
> so may need to fall under the same guidelines as RELEASE.
>
> Josh
>
> > On Jan 30, 2024, at 4:05 PM, Evan Jones <ev...@gmail.com> wrote:
> >
> > *When should this process begin? *In terms of where UserALE and currently
> > are.**
> > I want to get all *non-breaking* PRs merged into UserALE/test and
> > Distill/main and then do a new release of each of them, assuming there
> are
> > changes to release. UserALE has been queued up for a 2.4.0 for quite some
> > time. I don't think we have anything new for Distill. That way the
> monorepo
> > consolidation would happen on top of clean releases.
> >
> > I want to start the monorepo migration ASAP which means getting these
> > releases in progress ASAP.
> >
> > *The scope and sequence of the changes? Roadmap for the migration?*
> > If by scope you mean what aspects of the code base does this touch and
> > who's impacted, then the answer to the former is all aspects -- albeit
> > purely in a structural way -- and the answer to the latter is
> > contributors/devs only. Nothing about our individual projects will change
> > to start except they'll all be moved into the Flagon repo as
> > sub-directories. Within those directories, everything will be identical
> to
> > how they are currently. The only real repo that will see changes is the
> > Flagon repo.
> >
> > Because of this consolidation, the way developers contribute to the
> > projects will change. People will fork the Flagon Suite repo and merge
> back
> > into it regardless of the flagon product (distill, userale) they're
> > contributing to. Each project will still be released the same way.
> >
> > On the ASF side, we would have a single release process for the entire
> > Flagon project, rather than different releases for different
> sub-projects.
> > My hope is this vastly lowers overhead.
> >
> > The steps proposed above are the roadmap, more or less. As far as
> timeline
> > goes, I think the biggest bottleneck will be getting the current releases
> > out. I believe I can do the consolidation into a single repo in one
> weekend.
> >
> > *For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> others
> > that you might know of?*
> >
> > All documentation and website files would be stored in a single directory
> > in the repo. It sits alongside userale, distill, etc. The API
> documentation
> > for each project should leverage automated doc generators:
> >
> > - Distill = Sphinx + PyDoc --> ReadTheDocs
> > - UserALE = Sphinx + JsDoc --> ReadTheDocs
> >
> > The main flagon website should be the primary source of documentation for
> > things like tutorials, getting started, concept definitions, etc. It can
> > then point to the ReadTheDocs pages for each of packages' API documents.
> >
> > The overhauling of the documentation is not really couple to the monorepo
> > migration, though. The monorepo migration merely lowers to burden of the
> > workflow I describe above by centralizing everything.
> >
> > *I think we should set up a robust CI/CD pipeline to handle the monorepo
> to
> > ensure that all projects are integrated seamlessly and that automated
> tests
> > are in place for the monorepo.*
> >
> > Yes.
> >
> > Best
> >
> > Evan Jones
> > Website: www.ea-jones.com
> >
> >
> > On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <ag...@umd.edu> wrote:
> >
> >> Hi Evan,
> >>
> >> I support this monorepo structure - as you stated, there are
> >> several benefits of consolidating the Apache Flagon projects. Just a few
> >> questions/suggestions:
> >>
> >> - Do you have a timeline in mind for:
> >>
> >>
> >> - When should this process begin? *In terms of where UserALE and
> Distill
> >> currently are.*
> >> - The scope and sequence of the changes? Roadmap for the migration?
> >>
> >>
> >> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> >> others that you might know of?
> >> - I think we should set up a robust CI/CD pipeline to handle the
> >> monorepo to ensure that all projects are integrated seamlessly and
> that
> >> automated tests are in place for the monorepo.
> >>
> >>
> >> Best,
> >> *Amir M. Ghaemi*
> >>
> >> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com>
> >> wrote:
> >>
> >>> There are likely a few more steps:
> >>>
> >>> 1. Send a do not merge email for all repos
> >>> 2. Add userale/distill code, git tree, and github actions to the flagon
> >>> repo
> >>> 3. *Consolidate documents into a single repo*
> >>> 4. *Migrate to a build tool for monorepos like Bazel* (example:
> >>>
> https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
> >>> Beam uses gradle)
> >>> 5. Mark userale/distill repos as archived and update read me with a
> link
> >>> 6. Send a merge freely email
> >>> 7. Update flagon readme and flagon site links
> >>>
> >>> The bolded things could happen after the structural changes depending
> on
> >>> our priorities. If we implement those changes as part of the migration
> it
> >>> would simplify contribution conventions post migration but slow things
> >> down
> >>> in the short-term. If we implement them after, we'll open back up to
> >> merges
> >>> faster but the repo will be in a "dirty" state for longer as we
> >> transition
> >>> to monorepo best practices.
> >>>
> >>>> I don't think there's anything besides the flagon site that points to
> >> any
> >>> github repo, but I'm not sure.
> >>>
> >>> We definitely want to check our p's and q's though, especially on the
> >>> Apache side. Are the Distill, UserALE and Flagon repos treated as
> >> separate
> >>> projects / releases in the eyes of Apache or are their releases all
> >> bundled
> >>> into Flagon releases?
> >>>
> >>> Best
> >>>
> >>> Evan Jones
> >>> Website: www.ea-jones.com
> >>>
> >>>
> >>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
> >>>
> >>>> I'm all for a monorepo. Especially because it's hard to get visibility
> >>>> into a single project when its spread across multiple repos in the sea
> >> of
> >>>> the Apache GitHub org.
> >>>>
> >>>> Are these what the steps look like for moving to a monorepo?
> >>>>
> >>>> 1. Send a do not merge email for all repos
> >>>> 2. Add userale/distill code, git tree, and github actions to the
> flagon
> >>>> repo
> >>>> 3. Mark userale/distill repos as archived and update read me with a
> >> link
> >>>> 4. Send a merge freely email
> >>>> 5. Update flagon readme and flagon site links
> >>>>
> >>>> I don't think there's anything besides the flagon site that points to
> >> any
> >>>> github repo, but I'm not sure.
> >>>>
> >>>> On 2024/01/17 00:36:41 Evan Jones wrote:
> >>>>> Hi all,
> >>>>>
> >>>>> I've recently had more bandwidth to contribute to Flagon. Recently, I
> >>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up
> >> the
> >>>>> instrumentation team at ARLIS. My goal is for us to ramp up our
> >>>>> contribution to the Flagon suite.
> >>>>>
> >>>>> After spending quite a bit of time thinking about Flagon, I've come
> >> to
> >>>>> believe the best way for Flagon to deliver truly differentiated value
> >>> to
> >>>>> end users is to deliver on the original vision of all the
> >> sub-projects:
> >>>>>
> >>>>> - UserALE for instrumentation
> >>>>> - Distill for analytics
> >>>>> - STOUT as a UI for the analytics, metrics, and experiment
> >>>>> - TAP (re-envisioned) as the infrastructure platform to connect
> >> all
> >>>>> those pieces
> >>>>>
> >>>>> By eventually building all these pieces, Flagon could be a very
> >> strong
> >>>>> open-source alternative to services like Google Analytics, LogRocket,
> >>> and
> >>>>> others. I intend to socialize the above vision more in the future.
> >> For
> >>>> now
> >>>>> I digress since that's not the point of this email.
> >>>>>
> >>>>> The point of this email is to share an RFC
> >>>>> <
> >>>>
> >>>
> >>
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> >>>>>
> >>>>> for *an overhaul I believe needs to happen as a precursor to
> >> achieving
> >>>> the
> >>>>> above vision*.
> >>>>>
> >>>>> *Please share feedback, comments and concerns right on the google
> >> doc*.
> >>>>> I've also opened a GH issue <
> >>> https://github.com/apache/flagon/issues/44>
> >>>> if
> >>>>> folks would like to contribute to the conversation there, though I
> >>> prefer
> >>>>> to keep as much commentary within the document itself to simplify
> >>>> collation
> >>>>> later.
> >>>>>
> >>>>> Many thanks!
> >>>>>
> >>>>> Best
> >>>>>
> >>>>> Evan Jones
> >>>>>
> >>>>
> >>>
> >>
>
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Joshua Poore <po...@apache.org>.
The idea is growing on me—I’m pretty sure we can generate Pypi and NPM releases from the same repo, different folders.
Agree that now is the time to do it before user base on Distill grows.
I do think we should run a consensus/community VOTE on this. Let me refresh myself on which ruleset we should use for this. It’s a big change, so may need to fall under the same guidelines as RELEASE.
Josh
> On Jan 30, 2024, at 4:05 PM, Evan Jones <ev...@gmail.com> wrote:
>
> *When should this process begin? *In terms of where UserALE and currently
> are.**
> I want to get all *non-breaking* PRs merged into UserALE/test and
> Distill/main and then do a new release of each of them, assuming there are
> changes to release. UserALE has been queued up for a 2.4.0 for quite some
> time. I don't think we have anything new for Distill. That way the monorepo
> consolidation would happen on top of clean releases.
>
> I want to start the monorepo migration ASAP which means getting these
> releases in progress ASAP.
>
> *The scope and sequence of the changes? Roadmap for the migration?*
> If by scope you mean what aspects of the code base does this touch and
> who's impacted, then the answer to the former is all aspects -- albeit
> purely in a structural way -- and the answer to the latter is
> contributors/devs only. Nothing about our individual projects will change
> to start except they'll all be moved into the Flagon repo as
> sub-directories. Within those directories, everything will be identical to
> how they are currently. The only real repo that will see changes is the
> Flagon repo.
>
> Because of this consolidation, the way developers contribute to the
> projects will change. People will fork the Flagon Suite repo and merge back
> into it regardless of the flagon product (distill, userale) they're
> contributing to. Each project will still be released the same way.
>
> On the ASF side, we would have a single release process for the entire
> Flagon project, rather than different releases for different sub-projects.
> My hope is this vastly lowers overhead.
>
> The steps proposed above are the roadmap, more or less. As far as timeline
> goes, I think the biggest bottleneck will be getting the current releases
> out. I believe I can do the consolidation into a single repo in one weekend.
>
> *For documentation, do we intend to use ReadTheDocs and Sphinx? Any others
> that you might know of?*
>
> All documentation and website files would be stored in a single directory
> in the repo. It sits alongside userale, distill, etc. The API documentation
> for each project should leverage automated doc generators:
>
> - Distill = Sphinx + PyDoc --> ReadTheDocs
> - UserALE = Sphinx + JsDoc --> ReadTheDocs
>
> The main flagon website should be the primary source of documentation for
> things like tutorials, getting started, concept definitions, etc. It can
> then point to the ReadTheDocs pages for each of packages' API documents.
>
> The overhauling of the documentation is not really couple to the monorepo
> migration, though. The monorepo migration merely lowers to burden of the
> workflow I describe above by centralizing everything.
>
> *I think we should set up a robust CI/CD pipeline to handle the monorepo to
> ensure that all projects are integrated seamlessly and that automated tests
> are in place for the monorepo.*
>
> Yes.
>
> Best
>
> Evan Jones
> Website: www.ea-jones.com
>
>
> On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <ag...@umd.edu> wrote:
>
>> Hi Evan,
>>
>> I support this monorepo structure - as you stated, there are
>> several benefits of consolidating the Apache Flagon projects. Just a few
>> questions/suggestions:
>>
>> - Do you have a timeline in mind for:
>>
>>
>> - When should this process begin? *In terms of where UserALE and Distill
>> currently are.*
>> - The scope and sequence of the changes? Roadmap for the migration?
>>
>>
>> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any
>> others that you might know of?
>> - I think we should set up a robust CI/CD pipeline to handle the
>> monorepo to ensure that all projects are integrated seamlessly and that
>> automated tests are in place for the monorepo.
>>
>>
>> Best,
>> *Amir M. Ghaemi*
>>
>> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com>
>> wrote:
>>
>>> There are likely a few more steps:
>>>
>>> 1. Send a do not merge email for all repos
>>> 2. Add userale/distill code, git tree, and github actions to the flagon
>>> repo
>>> 3. *Consolidate documents into a single repo*
>>> 4. *Migrate to a build tool for monorepos like Bazel* (example:
>>> https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
>>> Beam uses gradle)
>>> 5. Mark userale/distill repos as archived and update read me with a link
>>> 6. Send a merge freely email
>>> 7. Update flagon readme and flagon site links
>>>
>>> The bolded things could happen after the structural changes depending on
>>> our priorities. If we implement those changes as part of the migration it
>>> would simplify contribution conventions post migration but slow things
>> down
>>> in the short-term. If we implement them after, we'll open back up to
>> merges
>>> faster but the repo will be in a "dirty" state for longer as we
>> transition
>>> to monorepo best practices.
>>>
>>>> I don't think there's anything besides the flagon site that points to
>> any
>>> github repo, but I'm not sure.
>>>
>>> We definitely want to check our p's and q's though, especially on the
>>> Apache side. Are the Distill, UserALE and Flagon repos treated as
>> separate
>>> projects / releases in the eyes of Apache or are their releases all
>> bundled
>>> into Flagon releases?
>>>
>>> Best
>>>
>>> Evan Jones
>>> Website: www.ea-jones.com
>>>
>>>
>>> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
>>>
>>>> I'm all for a monorepo. Especially because it's hard to get visibility
>>>> into a single project when its spread across multiple repos in the sea
>> of
>>>> the Apache GitHub org.
>>>>
>>>> Are these what the steps look like for moving to a monorepo?
>>>>
>>>> 1. Send a do not merge email for all repos
>>>> 2. Add userale/distill code, git tree, and github actions to the flagon
>>>> repo
>>>> 3. Mark userale/distill repos as archived and update read me with a
>> link
>>>> 4. Send a merge freely email
>>>> 5. Update flagon readme and flagon site links
>>>>
>>>> I don't think there's anything besides the flagon site that points to
>> any
>>>> github repo, but I'm not sure.
>>>>
>>>> On 2024/01/17 00:36:41 Evan Jones wrote:
>>>>> Hi all,
>>>>>
>>>>> I've recently had more bandwidth to contribute to Flagon. Recently, I
>>>>> helped with the release of Flagon-Distill v0.1.0 and am leading up
>> the
>>>>> instrumentation team at ARLIS. My goal is for us to ramp up our
>>>>> contribution to the Flagon suite.
>>>>>
>>>>> After spending quite a bit of time thinking about Flagon, I've come
>> to
>>>>> believe the best way for Flagon to deliver truly differentiated value
>>> to
>>>>> end users is to deliver on the original vision of all the
>> sub-projects:
>>>>>
>>>>> - UserALE for instrumentation
>>>>> - Distill for analytics
>>>>> - STOUT as a UI for the analytics, metrics, and experiment
>>>>> - TAP (re-envisioned) as the infrastructure platform to connect
>> all
>>>>> those pieces
>>>>>
>>>>> By eventually building all these pieces, Flagon could be a very
>> strong
>>>>> open-source alternative to services like Google Analytics, LogRocket,
>>> and
>>>>> others. I intend to socialize the above vision more in the future.
>> For
>>>> now
>>>>> I digress since that's not the point of this email.
>>>>>
>>>>> The point of this email is to share an RFC
>>>>> <
>>>>
>>>
>> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
>>>>>
>>>>> for *an overhaul I believe needs to happen as a precursor to
>> achieving
>>>> the
>>>>> above vision*.
>>>>>
>>>>> *Please share feedback, comments and concerns right on the google
>> doc*.
>>>>> I've also opened a GH issue <
>>> https://github.com/apache/flagon/issues/44>
>>>> if
>>>>> folks would like to contribute to the conversation there, though I
>>> prefer
>>>>> to keep as much commentary within the document itself to simplify
>>>> collation
>>>>> later.
>>>>>
>>>>> Many thanks!
>>>>>
>>>>> Best
>>>>>
>>>>> Evan Jones
>>>>>
>>>>
>>>
>>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Evan Jones <ev...@gmail.com>.
*When should this process begin? *In terms of where UserALE and currently
are.**
I want to get all *non-breaking* PRs merged into UserALE/test and
Distill/main and then do a new release of each of them, assuming there are
changes to release. UserALE has been queued up for a 2.4.0 for quite some
time. I don't think we have anything new for Distill. That way the monorepo
consolidation would happen on top of clean releases.
I want to start the monorepo migration ASAP which means getting these
releases in progress ASAP.
*The scope and sequence of the changes? Roadmap for the migration?*
If by scope you mean what aspects of the code base does this touch and
who's impacted, then the answer to the former is all aspects -- albeit
purely in a structural way -- and the answer to the latter is
contributors/devs only. Nothing about our individual projects will change
to start except they'll all be moved into the Flagon repo as
sub-directories. Within those directories, everything will be identical to
how they are currently. The only real repo that will see changes is the
Flagon repo.
Because of this consolidation, the way developers contribute to the
projects will change. People will fork the Flagon Suite repo and merge back
into it regardless of the flagon product (distill, userale) they're
contributing to. Each project will still be released the same way.
On the ASF side, we would have a single release process for the entire
Flagon project, rather than different releases for different sub-projects.
My hope is this vastly lowers overhead.
The steps proposed above are the roadmap, more or less. As far as timeline
goes, I think the biggest bottleneck will be getting the current releases
out. I believe I can do the consolidation into a single repo in one weekend.
*For documentation, do we intend to use ReadTheDocs and Sphinx? Any others
that you might know of?*
All documentation and website files would be stored in a single directory
in the repo. It sits alongside userale, distill, etc. The API documentation
for each project should leverage automated doc generators:
- Distill = Sphinx + PyDoc --> ReadTheDocs
- UserALE = Sphinx + JsDoc --> ReadTheDocs
The main flagon website should be the primary source of documentation for
things like tutorials, getting started, concept definitions, etc. It can
then point to the ReadTheDocs pages for each of packages' API documents.
The overhauling of the documentation is not really couple to the monorepo
migration, though. The monorepo migration merely lowers to burden of the
workflow I describe above by centralizing everything.
*I think we should set up a robust CI/CD pipeline to handle the monorepo to
ensure that all projects are integrated seamlessly and that automated tests
are in place for the monorepo.*
Yes.
Best
Evan Jones
Website: www.ea-jones.com
On Tue, Jan 30, 2024 at 2:21 PM Amir Ghaemi <ag...@umd.edu> wrote:
> Hi Evan,
>
> I support this monorepo structure - as you stated, there are
> several benefits of consolidating the Apache Flagon projects. Just a few
> questions/suggestions:
>
> - Do you have a timeline in mind for:
>
>
> - When should this process begin? *In terms of where UserALE and Distill
> currently are.*
> - The scope and sequence of the changes? Roadmap for the migration?
>
>
> - For documentation, do we intend to use ReadTheDocs and Sphinx? Any
> others that you might know of?
> - I think we should set up a robust CI/CD pipeline to handle the
> monorepo to ensure that all projects are integrated seamlessly and that
> automated tests are in place for the monorepo.
>
>
> Best,
> *Amir M. Ghaemi*
>
> On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com>
> wrote:
>
> > There are likely a few more steps:
> >
> > 1. Send a do not merge email for all repos
> > 2. Add userale/distill code, git tree, and github actions to the flagon
> > repo
> > 3. *Consolidate documents into a single repo*
> > 4. *Migrate to a build tool for monorepos like Bazel* (example:
> > https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
> > Beam uses gradle)
> > 5. Mark userale/distill repos as archived and update read me with a link
> > 6. Send a merge freely email
> > 7. Update flagon readme and flagon site links
> >
> > The bolded things could happen after the structural changes depending on
> > our priorities. If we implement those changes as part of the migration it
> > would simplify contribution conventions post migration but slow things
> down
> > in the short-term. If we implement them after, we'll open back up to
> merges
> > faster but the repo will be in a "dirty" state for longer as we
> transition
> > to monorepo best practices.
> >
> > > I don't think there's anything besides the flagon site that points to
> any
> > github repo, but I'm not sure.
> >
> > We definitely want to check our p's and q's though, especially on the
> > Apache side. Are the Distill, UserALE and Flagon repos treated as
> separate
> > projects / releases in the eyes of Apache or are their releases all
> bundled
> > into Flagon releases?
> >
> > Best
> >
> > Evan Jones
> > Website: www.ea-jones.com
> >
> >
> > On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
> >
> > > I'm all for a monorepo. Especially because it's hard to get visibility
> > > into a single project when its spread across multiple repos in the sea
> of
> > > the Apache GitHub org.
> > >
> > > Are these what the steps look like for moving to a monorepo?
> > >
> > > 1. Send a do not merge email for all repos
> > > 2. Add userale/distill code, git tree, and github actions to the flagon
> > > repo
> > > 3. Mark userale/distill repos as archived and update read me with a
> link
> > > 4. Send a merge freely email
> > > 5. Update flagon readme and flagon site links
> > >
> > > I don't think there's anything besides the flagon site that points to
> any
> > > github repo, but I'm not sure.
> > >
> > > On 2024/01/17 00:36:41 Evan Jones wrote:
> > > > Hi all,
> > > >
> > > > I've recently had more bandwidth to contribute to Flagon. Recently, I
> > > > helped with the release of Flagon-Distill v0.1.0 and am leading up
> the
> > > > instrumentation team at ARLIS. My goal is for us to ramp up our
> > > > contribution to the Flagon suite.
> > > >
> > > > After spending quite a bit of time thinking about Flagon, I've come
> to
> > > > believe the best way for Flagon to deliver truly differentiated value
> > to
> > > > end users is to deliver on the original vision of all the
> sub-projects:
> > > >
> > > > - UserALE for instrumentation
> > > > - Distill for analytics
> > > > - STOUT as a UI for the analytics, metrics, and experiment
> > > > - TAP (re-envisioned) as the infrastructure platform to connect
> all
> > > > those pieces
> > > >
> > > > By eventually building all these pieces, Flagon could be a very
> strong
> > > > open-source alternative to services like Google Analytics, LogRocket,
> > and
> > > > others. I intend to socialize the above vision more in the future.
> For
> > > now
> > > > I digress since that's not the point of this email.
> > > >
> > > > The point of this email is to share an RFC
> > > > <
> > >
> >
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> > > >
> > > > for *an overhaul I believe needs to happen as a precursor to
> achieving
> > > the
> > > > above vision*.
> > > >
> > > > *Please share feedback, comments and concerns right on the google
> doc*.
> > > > I've also opened a GH issue <
> > https://github.com/apache/flagon/issues/44>
> > > if
> > > > folks would like to contribute to the conversation there, though I
> > prefer
> > > > to keep as much commentary within the document itself to simplify
> > > collation
> > > > later.
> > > >
> > > > Many thanks!
> > > >
> > > > Best
> > > >
> > > > Evan Jones
> > > >
> > >
> >
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Amir Ghaemi <ag...@umd.edu>.
Hi Evan,
I support this monorepo structure - as you stated, there are
several benefits of consolidating the Apache Flagon projects. Just a few
questions/suggestions:
- Do you have a timeline in mind for:
- When should this process begin? *In terms of where UserALE and Distill
currently are.*
- The scope and sequence of the changes? Roadmap for the migration?
- For documentation, do we intend to use ReadTheDocs and Sphinx? Any
others that you might know of?
- I think we should set up a robust CI/CD pipeline to handle the
monorepo to ensure that all projects are integrated seamlessly and that
automated tests are in place for the monorepo.
Best,
*Amir M. Ghaemi*
On Thu, Jan 25, 2024 at 3:01 PM Evan Jones <ev...@gmail.com> wrote:
> There are likely a few more steps:
>
> 1. Send a do not merge email for all repos
> 2. Add userale/distill code, git tree, and github actions to the flagon
> repo
> 3. *Consolidate documents into a single repo*
> 4. *Migrate to a build tool for monorepos like Bazel* (example:
> https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
> Beam uses gradle)
> 5. Mark userale/distill repos as archived and update read me with a link
> 6. Send a merge freely email
> 7. Update flagon readme and flagon site links
>
> The bolded things could happen after the structural changes depending on
> our priorities. If we implement those changes as part of the migration it
> would simplify contribution conventions post migration but slow things down
> in the short-term. If we implement them after, we'll open back up to merges
> faster but the repo will be in a "dirty" state for longer as we transition
> to monorepo best practices.
>
> > I don't think there's anything besides the flagon site that points to any
> github repo, but I'm not sure.
>
> We definitely want to check our p's and q's though, especially on the
> Apache side. Are the Distill, UserALE and Flagon repos treated as separate
> projects / releases in the eyes of Apache or are their releases all bundled
> into Flagon releases?
>
> Best
>
> Evan Jones
> Website: www.ea-jones.com
>
>
> On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
>
> > I'm all for a monorepo. Especially because it's hard to get visibility
> > into a single project when its spread across multiple repos in the sea of
> > the Apache GitHub org.
> >
> > Are these what the steps look like for moving to a monorepo?
> >
> > 1. Send a do not merge email for all repos
> > 2. Add userale/distill code, git tree, and github actions to the flagon
> > repo
> > 3. Mark userale/distill repos as archived and update read me with a link
> > 4. Send a merge freely email
> > 5. Update flagon readme and flagon site links
> >
> > I don't think there's anything besides the flagon site that points to any
> > github repo, but I'm not sure.
> >
> > On 2024/01/17 00:36:41 Evan Jones wrote:
> > > Hi all,
> > >
> > > I've recently had more bandwidth to contribute to Flagon. Recently, I
> > > helped with the release of Flagon-Distill v0.1.0 and am leading up the
> > > instrumentation team at ARLIS. My goal is for us to ramp up our
> > > contribution to the Flagon suite.
> > >
> > > After spending quite a bit of time thinking about Flagon, I've come to
> > > believe the best way for Flagon to deliver truly differentiated value
> to
> > > end users is to deliver on the original vision of all the sub-projects:
> > >
> > > - UserALE for instrumentation
> > > - Distill for analytics
> > > - STOUT as a UI for the analytics, metrics, and experiment
> > > - TAP (re-envisioned) as the infrastructure platform to connect all
> > > those pieces
> > >
> > > By eventually building all these pieces, Flagon could be a very strong
> > > open-source alternative to services like Google Analytics, LogRocket,
> and
> > > others. I intend to socialize the above vision more in the future. For
> > now
> > > I digress since that's not the point of this email.
> > >
> > > The point of this email is to share an RFC
> > > <
> >
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> > >
> > > for *an overhaul I believe needs to happen as a precursor to achieving
> > the
> > > above vision*.
> > >
> > > *Please share feedback, comments and concerns right on the google doc*.
> > > I've also opened a GH issue <
> https://github.com/apache/flagon/issues/44>
> > if
> > > folks would like to contribute to the conversation there, though I
> prefer
> > > to keep as much commentary within the document itself to simplify
> > collation
> > > later.
> > >
> > > Many thanks!
> > >
> > > Best
> > >
> > > Evan Jones
> > >
> >
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Evan Jones <ev...@gmail.com>.
There are likely a few more steps:
1. Send a do not merge email for all repos
2. Add userale/distill code, git tree, and github actions to the flagon repo
3. *Consolidate documents into a single repo*
4. *Migrate to a build tool for monorepos like Bazel* (example:
https://github.com/thundergolfer/example-bazel-monorepo/tree/master/cli;
Beam uses gradle)
5. Mark userale/distill repos as archived and update read me with a link
6. Send a merge freely email
7. Update flagon readme and flagon site links
The bolded things could happen after the structural changes depending on
our priorities. If we implement those changes as part of the migration it
would simplify contribution conventions post migration but slow things down
in the short-term. If we implement them after, we'll open back up to merges
faster but the repo will be in a "dirty" state for longer as we transition
to monorepo best practices.
> I don't think there's anything besides the flagon site that points to any
github repo, but I'm not sure.
We definitely want to check our p's and q's though, especially on the
Apache side. Are the Distill, UserALE and Flagon repos treated as separate
projects / releases in the eyes of Apache or are their releases all bundled
into Flagon releases?
Best
Evan Jones
Website: www.ea-jones.com
On Thu, Jan 25, 2024 at 10:40 AM Jason Young <jk...@apache.org> wrote:
> I'm all for a monorepo. Especially because it's hard to get visibility
> into a single project when its spread across multiple repos in the sea of
> the Apache GitHub org.
>
> Are these what the steps look like for moving to a monorepo?
>
> 1. Send a do not merge email for all repos
> 2. Add userale/distill code, git tree, and github actions to the flagon
> repo
> 3. Mark userale/distill repos as archived and update read me with a link
> 4. Send a merge freely email
> 5. Update flagon readme and flagon site links
>
> I don't think there's anything besides the flagon site that points to any
> github repo, but I'm not sure.
>
> On 2024/01/17 00:36:41 Evan Jones wrote:
> > Hi all,
> >
> > I've recently had more bandwidth to contribute to Flagon. Recently, I
> > helped with the release of Flagon-Distill v0.1.0 and am leading up the
> > instrumentation team at ARLIS. My goal is for us to ramp up our
> > contribution to the Flagon suite.
> >
> > After spending quite a bit of time thinking about Flagon, I've come to
> > believe the best way for Flagon to deliver truly differentiated value to
> > end users is to deliver on the original vision of all the sub-projects:
> >
> > - UserALE for instrumentation
> > - Distill for analytics
> > - STOUT as a UI for the analytics, metrics, and experiment
> > - TAP (re-envisioned) as the infrastructure platform to connect all
> > those pieces
> >
> > By eventually building all these pieces, Flagon could be a very strong
> > open-source alternative to services like Google Analytics, LogRocket, and
> > others. I intend to socialize the above vision more in the future. For
> now
> > I digress since that's not the point of this email.
> >
> > The point of this email is to share an RFC
> > <
> https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing
> >
> > for *an overhaul I believe needs to happen as a precursor to achieving
> the
> > above vision*.
> >
> > *Please share feedback, comments and concerns right on the google doc*.
> > I've also opened a GH issue <https://github.com/apache/flagon/issues/44>
> if
> > folks would like to contribute to the conversation there, though I prefer
> > to keep as much commentary within the document itself to simplify
> collation
> > later.
> >
> > Many thanks!
> >
> > Best
> >
> > Evan Jones
> >
>
Re: [RFC] Consolidation of Apache Flagon Projects into a Monorepo Structure
Posted by Jason Young <jk...@apache.org>.
I'm all for a monorepo. Especially because it's hard to get visibility into a single project when its spread across multiple repos in the sea of the Apache GitHub org.
Are these what the steps look like for moving to a monorepo?
1. Send a do not merge email for all repos
2. Add userale/distill code, git tree, and github actions to the flagon repo
3. Mark userale/distill repos as archived and update read me with a link
4. Send a merge freely email
5. Update flagon readme and flagon site links
I don't think there's anything besides the flagon site that points to any github repo, but I'm not sure.
On 2024/01/17 00:36:41 Evan Jones wrote:
> Hi all,
>
> I've recently had more bandwidth to contribute to Flagon. Recently, I
> helped with the release of Flagon-Distill v0.1.0 and am leading up the
> instrumentation team at ARLIS. My goal is for us to ramp up our
> contribution to the Flagon suite.
>
> After spending quite a bit of time thinking about Flagon, I've come to
> believe the best way for Flagon to deliver truly differentiated value to
> end users is to deliver on the original vision of all the sub-projects:
>
> - UserALE for instrumentation
> - Distill for analytics
> - STOUT as a UI for the analytics, metrics, and experiment
> - TAP (re-envisioned) as the infrastructure platform to connect all
> those pieces
>
> By eventually building all these pieces, Flagon could be a very strong
> open-source alternative to services like Google Analytics, LogRocket, and
> others. I intend to socialize the above vision more in the future. For now
> I digress since that's not the point of this email.
>
> The point of this email is to share an RFC
> <https://docs.google.com/document/d/1xxZDchgjLswbzWpVz0GcRlLqLCEygW_cTvcLhsombtk/edit?usp=sharing>
> for *an overhaul I believe needs to happen as a precursor to achieving the
> above vision*.
>
> *Please share feedback, comments and concerns right on the google doc*.
> I've also opened a GH issue <https://github.com/apache/flagon/issues/44> if
> folks would like to contribute to the conversation there, though I prefer
> to keep as much commentary within the document itself to simplify collation
> later.
>
> Many thanks!
>
> Best
>
> Evan Jones
>