You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@streams.apache.org by "sblackmon@apache.org" <sb...@apache.org> on 2023/12/04 02:21:06 UTC
New Initiative
Hello dev@streams.a.o,
This list has been very quiet for a very long time. I’m hoping to change that, and generate some momentum toward a worth-while initiative right in-line with Streams founding goals.
The project founding proposal [1] includes the following goals:
• Publication of Activities from multiple systems via HTTP
• Aggregation and syndication of streams
• Support for security trimming of streams by social graph
• Noise reduction and intelligent filtering
• Federation of streams across disparate systems
• Provide libraries for easy integration in source systems
The code we actually built focused most heavily on goals #2 and #5, and evolved into a (useful IMO, at least around the time of our TLP graduation) java SDK for building complex social media application back-end ’streams’. But, we never shipped software with clear stand-alone utility for anyone other than java developers. The internet, and the social media landscape specifically, has changed a lot since then, and I would argue that the system envisioned in the proposal is needed now more than ever. I won’t go right now into all the reasons I believe that is the case, because it would be TL;DR and off-topic, but will just say that there appears to be an unprecedented (at least in this century) interest in hosting social-media-like software that can inter-operate across installations.
So I’m here asking whether anyone is open to a discussion about re-aligning this project to the totality of it’s origin, with a new North Star:
• a containerized deployable software stack that provides an Activity Streams 2.0 / Activity Pub interface via rest API
• Confirmed interoperable with the Mastodon front-end, and other ActivityPub client software
• Built to scale with minimal system admin burden to the needs of the largest existing Mastodon instances: ballpark ~1M total users, 10K concurrent
• Able to scale far beyond that for an experienced team: a medium-sized Web 2.0 social network: ~100M total users, up to ~1M concurrent
I don’t think it’s hyper-bole to observe that the future of social media is in flux, and in early stages of a 10+ year trend toward decentralization. I for want really want that decentralized future to be based on Activity Streams 2.0 and ActivityPub, now official W3C standards, and thus inter-operable when desired, rather than completely silo’ed like the prior generation. Also I think the above is entirely plausible on a 1-2 year time-scale.
Does anyone else agree? Is anyone willing to support a pivot like this, and try to re-build this product and community with me?
• [1] https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
Steve Blackmon
Apache Streams PMC Chair
Re: New Initiative
Posted by Steve Blackmon <sb...@apache.org>.
There may be projects that have entirely migrated away from Jenkins, but
many perhaps the majority appear to still be using it:
https://ci-builds.apache.org/
I got our GH Actions working again (build only):
https://github.com/apache/streams/actions/workflows/streams-ci-workflow.yml
And got Jenkins building Streams as well (JDK17 only):
https://ci-builds.apache.org/job/Streams/job/pipeline/job/master/
Unlike previously, the build is governed by a Jenkinsfile:
https://github.com/apache/streams/blob/master/Jenkinsfile
Historically Jenkins deployed Streams SNAPSHOTS on every build, but that is
not the case right now b/c I got 401s when 'deploy' was added to list of
goals.
Steve
On Mon, Dec 18, 2023 at 9:37 AM Trevor Grant <tr...@gmail.com>
wrote:
> iirc asf-infra deactivated jenkins all together in favor of gh actions.
>
> On Mon, Dec 18, 2023 at 9:35 AM Steve Blackmon <sb...@apache.org>
> wrote:
>
> > Closing the loop on vulnerable dependencies, I wound up (for now)
> > deactivating streams-persist-hdfs and streams-persist-elasticsearch, and
> > all of the sub-modules of streams-examples. Suneel I think you are right
> > opensearch is a more appropriate dependency if there is a reason to bring
> > streams-persist-elasticsearch back into the reactor.
> >
> > Build and unit tests were working for me in JDK8, JDK11, and JDK17. Our
> > github actions CI needs some love and i suspect our jenkins jobs got lost
> > during an infra upgrade so we'll need to bring that back online. Those
> > details aside, I think the project is release-able again.
> >
> > I have not yet typed up anything more than the first message in this
> > thread. I'm planning to spend time over the holiday digging into the
> > current revisions of the Activity Streams 2.0 ontology and ActivityPub
> > spec, and how much of them are currently supported by the standard
> mastodon
> > back-end. If anyone else wants to help out in this early stage, maybe do
> > the same and we can all compare notes on what the interfaces and
> deployment
> > plan of a non-throwaway-proof-of-concept/MVP might look like?
> >
> > Just riffing, I suspect we will want to continue trimming down modules
> and
> > dependencies if they don't have a clear purpose under the new objective
> of
> > a scalable activitypub backend plus useful plugins (such as the
> persistence
> > modules, data providers, and data augmentation modules in
> > streams-contrib). I think we could approach a form where our core
> > dependencies are apache projects only - commons, httpcomponents, juneau,
> > lucene/solr, maybe pekko? There will need to be at least one default
> data
> > repository capable of storing anything in the spec and supporting the
> > queries required by any endpoint. Jena seems like a good candidate for
> > this, at least for small-ish deployments, but there might be another
> better
> > solution I've not thought of yet.
> >
> > P.S. It's nice to hear from everyone, I hope you are all doing well.
> >
> > On Sun, Dec 17, 2023 at 9:10 PM Brian Hodge <br...@gmail.com>
> wrote:
> >
> > > Apologies, this got buried in my inbox under other mailing lists.
> > >
> > > Steve, your suggestions make sense to me. I strongly agree that
> building
> > on
> > > Streams to ship broadly useful software is a worthwhile goal.
> > >
> > > I do have some questions about the realignment you're envisioning. Have
> > you
> > > written (or do you plan to write) anything more longform on design /
> > > architecture of the refactor you have in mind?
> > >
> > > I'm happy to help however I can. Are there near-term tasks that I and
> > > others can pitch in on?
> > >
> > > -Brian Hodge
> > >
> > > On Sun, Dec 17, 2023 at 2:58 PM Suneel Marthi <sm...@apache.org>
> > wrote:
> > >
> > > > Replace ElasticSearch with OpenSearch ???
> > > >
> > > > On Sun, Dec 17, 2023 at 12:36 PM Steve Blackmon <
> sblackmon@apache.org>
> > > > wrote:
> > > >
> > > > > Thanks Trevor. Silent majority, hope to hear from you as/if the
> > reboot
> > > > > picks up steam.
> > > > >
> > > > > I've gotten started the last few weeks by refactoring to enable
> > builds
> > > > with
> > > > > Java 17 (committed) and a purge of vulnerable dependencies (almost
> > > > ready).
> > > > >
> > > > > It looks like most of of modules will be able to remain active in
> the
> > > > > reactor to achieve the above, possible exceptions of our hdfs and
> > > > > elasticsearch modules. Elasticsearch is very outdated and even the
> > > > newest
> > > > > available common-hdfs doesn't appear able to have the vulnerable
> > > > > commons-collections:commons-collections removed and still work.
> > > > >
> > > > > Stay tuned
> > > > >
> > > > > On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <
> > trevor.d.grant@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > I agree with most of what you're saying, candidly my bet is on
> > social
> > > > > media
> > > > > > as we know it just sort of silently fading as Millenials get
> older.
> > > But
> > > > > I'm
> > > > > > still on board to help out- I have cycles to work tickets, but
> idk
> > > if I
> > > > > > have enough to put in any good faith effort to lead (we're doing
> a
> > > > > similar
> > > > > > reboot over on Mahout, where I am leading).
> > > > > >
> > > > > > tg
> > > > > >
> > > > > > On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <
> > > > > sblackmon@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Hello dev@streams.a.o,
> > > > > > >
> > > > > > > This list has been very quiet for a very long time. I’m hoping
> > to
> > > > > change
> > > > > > > that, and generate some momentum toward a worth-while
> initiative
> > > > right
> > > > > > > in-line with Streams founding goals.
> > > > > > >
> > > > > > > The project founding proposal [1] includes the following goals:
> > > > > > >
> > > > > > > • Publication of Activities from multiple systems via HTTP
> > > > > > > • Aggregation and syndication of streams
> > > > > > > • Support for security trimming of streams by social graph
> > > > > > > • Noise reduction and intelligent filtering
> > > > > > > • Federation of streams across disparate systems
> > > > > > > • Provide libraries for easy integration in source systems
> > > > > > >
> > > > > > > The code we actually built focused most heavily on goals #2 and
> > #5,
> > > > and
> > > > > > > evolved into a (useful IMO, at least around the time of our TLP
> > > > > > graduation)
> > > > > > > java SDK for building complex social media application back-end
> > > > > > > ’streams’. But, we never shipped software with clear
> stand-alone
> > > > > utility
> > > > > > > for anyone other than java developers. The internet, and the
> > > social
> > > > > > media
> > > > > > > landscape specifically, has changed a lot since then, and I
> would
> > > > argue
> > > > > > > that the system envisioned in the proposal is needed now more
> > than
> > > > > > ever. I
> > > > > > > won’t go right now into all the reasons I believe that is the
> > case,
> > > > > > because
> > > > > > > it would be TL;DR and off-topic, but will just say that there
> > > appears
> > > > > to
> > > > > > be
> > > > > > > an unprecedented (at least in this century) interest in hosting
> > > > > > > social-media-like software that can inter-operate across
> > > > installations.
> > > > > > >
> > > > > > > So I’m here asking whether anyone is open to a discussion about
> > > > > > > re-aligning this project to the totality of it’s origin, with a
> > new
> > > > > North
> > > > > > > Star:
> > > > > > >
> > > > > > > • a containerized deployable software stack that provides an
> > > Activity
> > > > > > > Streams 2.0 / Activity Pub interface via rest API
> > > > > > > • Confirmed interoperable with the Mastodon front-end, and
> other
> > > > > > > ActivityPub client software
> > > > > > > • Built to scale with minimal system admin burden to the needs
> of
> > > the
> > > > > > > largest existing Mastodon instances: ballpark ~1M total users,
> > 10K
> > > > > > > concurrent
> > > > > > > • Able to scale far beyond that for an experienced team: a
> > > > medium-sized
> > > > > > > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> > > > > > >
> > > > > > > I don’t think it’s hyper-bole to observe that the future of
> > social
> > > > > media
> > > > > > > is in flux, and in early stages of a 10+ year trend toward
> > > > > > > decentralization. I for want really want that decentralized
> > future
> > > > to
> > > > > be
> > > > > > > based on Activity Streams 2.0 and ActivityPub, now official W3C
> > > > > > standards,
> > > > > > > and thus inter-operable when desired, rather than completely
> > > silo’ed
> > > > > like
> > > > > > > the prior generation. Also I think the above is entirely
> > plausible
> > > > on
> > > > > a
> > > > > > > 1-2 year time-scale.
> > > > > > >
> > > > > > > Does anyone else agree? Is anyone willing to support a pivot
> > like
> > > > > this,
> > > > > > > and try to re-build this product and community with me?
> > > > > > >
> > > > > > > • [1]
> > > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> > > > > > >
> > > > > > >
> > > > > > > Steve Blackmon
> > > > > > > Apache Streams PMC Chair
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Initiative
Posted by Trevor Grant <tr...@gmail.com>.
iirc asf-infra deactivated jenkins all together in favor of gh actions.
On Mon, Dec 18, 2023 at 9:35 AM Steve Blackmon <sb...@apache.org> wrote:
> Closing the loop on vulnerable dependencies, I wound up (for now)
> deactivating streams-persist-hdfs and streams-persist-elasticsearch, and
> all of the sub-modules of streams-examples. Suneel I think you are right
> opensearch is a more appropriate dependency if there is a reason to bring
> streams-persist-elasticsearch back into the reactor.
>
> Build and unit tests were working for me in JDK8, JDK11, and JDK17. Our
> github actions CI needs some love and i suspect our jenkins jobs got lost
> during an infra upgrade so we'll need to bring that back online. Those
> details aside, I think the project is release-able again.
>
> I have not yet typed up anything more than the first message in this
> thread. I'm planning to spend time over the holiday digging into the
> current revisions of the Activity Streams 2.0 ontology and ActivityPub
> spec, and how much of them are currently supported by the standard mastodon
> back-end. If anyone else wants to help out in this early stage, maybe do
> the same and we can all compare notes on what the interfaces and deployment
> plan of a non-throwaway-proof-of-concept/MVP might look like?
>
> Just riffing, I suspect we will want to continue trimming down modules and
> dependencies if they don't have a clear purpose under the new objective of
> a scalable activitypub backend plus useful plugins (such as the persistence
> modules, data providers, and data augmentation modules in
> streams-contrib). I think we could approach a form where our core
> dependencies are apache projects only - commons, httpcomponents, juneau,
> lucene/solr, maybe pekko? There will need to be at least one default data
> repository capable of storing anything in the spec and supporting the
> queries required by any endpoint. Jena seems like a good candidate for
> this, at least for small-ish deployments, but there might be another better
> solution I've not thought of yet.
>
> P.S. It's nice to hear from everyone, I hope you are all doing well.
>
> On Sun, Dec 17, 2023 at 9:10 PM Brian Hodge <br...@gmail.com> wrote:
>
> > Apologies, this got buried in my inbox under other mailing lists.
> >
> > Steve, your suggestions make sense to me. I strongly agree that building
> on
> > Streams to ship broadly useful software is a worthwhile goal.
> >
> > I do have some questions about the realignment you're envisioning. Have
> you
> > written (or do you plan to write) anything more longform on design /
> > architecture of the refactor you have in mind?
> >
> > I'm happy to help however I can. Are there near-term tasks that I and
> > others can pitch in on?
> >
> > -Brian Hodge
> >
> > On Sun, Dec 17, 2023 at 2:58 PM Suneel Marthi <sm...@apache.org>
> wrote:
> >
> > > Replace ElasticSearch with OpenSearch ???
> > >
> > > On Sun, Dec 17, 2023 at 12:36 PM Steve Blackmon <sb...@apache.org>
> > > wrote:
> > >
> > > > Thanks Trevor. Silent majority, hope to hear from you as/if the
> reboot
> > > > picks up steam.
> > > >
> > > > I've gotten started the last few weeks by refactoring to enable
> builds
> > > with
> > > > Java 17 (committed) and a purge of vulnerable dependencies (almost
> > > ready).
> > > >
> > > > It looks like most of of modules will be able to remain active in the
> > > > reactor to achieve the above, possible exceptions of our hdfs and
> > > > elasticsearch modules. Elasticsearch is very outdated and even the
> > > newest
> > > > available common-hdfs doesn't appear able to have the vulnerable
> > > > commons-collections:commons-collections removed and still work.
> > > >
> > > > Stay tuned
> > > >
> > > > On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <
> trevor.d.grant@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > I agree with most of what you're saying, candidly my bet is on
> social
> > > > media
> > > > > as we know it just sort of silently fading as Millenials get older.
> > But
> > > > I'm
> > > > > still on board to help out- I have cycles to work tickets, but idk
> > if I
> > > > > have enough to put in any good faith effort to lead (we're doing a
> > > > similar
> > > > > reboot over on Mahout, where I am leading).
> > > > >
> > > > > tg
> > > > >
> > > > > On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <
> > > > sblackmon@apache.org>
> > > > > wrote:
> > > > >
> > > > > > Hello dev@streams.a.o,
> > > > > >
> > > > > > This list has been very quiet for a very long time. I’m hoping
> to
> > > > change
> > > > > > that, and generate some momentum toward a worth-while initiative
> > > right
> > > > > > in-line with Streams founding goals.
> > > > > >
> > > > > > The project founding proposal [1] includes the following goals:
> > > > > >
> > > > > > • Publication of Activities from multiple systems via HTTP
> > > > > > • Aggregation and syndication of streams
> > > > > > • Support for security trimming of streams by social graph
> > > > > > • Noise reduction and intelligent filtering
> > > > > > • Federation of streams across disparate systems
> > > > > > • Provide libraries for easy integration in source systems
> > > > > >
> > > > > > The code we actually built focused most heavily on goals #2 and
> #5,
> > > and
> > > > > > evolved into a (useful IMO, at least around the time of our TLP
> > > > > graduation)
> > > > > > java SDK for building complex social media application back-end
> > > > > > ’streams’. But, we never shipped software with clear stand-alone
> > > > utility
> > > > > > for anyone other than java developers. The internet, and the
> > social
> > > > > media
> > > > > > landscape specifically, has changed a lot since then, and I would
> > > argue
> > > > > > that the system envisioned in the proposal is needed now more
> than
> > > > > ever. I
> > > > > > won’t go right now into all the reasons I believe that is the
> case,
> > > > > because
> > > > > > it would be TL;DR and off-topic, but will just say that there
> > appears
> > > > to
> > > > > be
> > > > > > an unprecedented (at least in this century) interest in hosting
> > > > > > social-media-like software that can inter-operate across
> > > installations.
> > > > > >
> > > > > > So I’m here asking whether anyone is open to a discussion about
> > > > > > re-aligning this project to the totality of it’s origin, with a
> new
> > > > North
> > > > > > Star:
> > > > > >
> > > > > > • a containerized deployable software stack that provides an
> > Activity
> > > > > > Streams 2.0 / Activity Pub interface via rest API
> > > > > > • Confirmed interoperable with the Mastodon front-end, and other
> > > > > > ActivityPub client software
> > > > > > • Built to scale with minimal system admin burden to the needs of
> > the
> > > > > > largest existing Mastodon instances: ballpark ~1M total users,
> 10K
> > > > > > concurrent
> > > > > > • Able to scale far beyond that for an experienced team: a
> > > medium-sized
> > > > > > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> > > > > >
> > > > > > I don’t think it’s hyper-bole to observe that the future of
> social
> > > > media
> > > > > > is in flux, and in early stages of a 10+ year trend toward
> > > > > > decentralization. I for want really want that decentralized
> future
> > > to
> > > > be
> > > > > > based on Activity Streams 2.0 and ActivityPub, now official W3C
> > > > > standards,
> > > > > > and thus inter-operable when desired, rather than completely
> > silo’ed
> > > > like
> > > > > > the prior generation. Also I think the above is entirely
> plausible
> > > on
> > > > a
> > > > > > 1-2 year time-scale.
> > > > > >
> > > > > > Does anyone else agree? Is anyone willing to support a pivot
> like
> > > > this,
> > > > > > and try to re-build this product and community with me?
> > > > > >
> > > > > > • [1]
> > > > > >
> > > https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> > > > > >
> > > > > >
> > > > > > Steve Blackmon
> > > > > > Apache Streams PMC Chair
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: New Initiative
Posted by Steve Blackmon <sb...@apache.org>.
Closing the loop on vulnerable dependencies, I wound up (for now)
deactivating streams-persist-hdfs and streams-persist-elasticsearch, and
all of the sub-modules of streams-examples. Suneel I think you are right
opensearch is a more appropriate dependency if there is a reason to bring
streams-persist-elasticsearch back into the reactor.
Build and unit tests were working for me in JDK8, JDK11, and JDK17. Our
github actions CI needs some love and i suspect our jenkins jobs got lost
during an infra upgrade so we'll need to bring that back online. Those
details aside, I think the project is release-able again.
I have not yet typed up anything more than the first message in this
thread. I'm planning to spend time over the holiday digging into the
current revisions of the Activity Streams 2.0 ontology and ActivityPub
spec, and how much of them are currently supported by the standard mastodon
back-end. If anyone else wants to help out in this early stage, maybe do
the same and we can all compare notes on what the interfaces and deployment
plan of a non-throwaway-proof-of-concept/MVP might look like?
Just riffing, I suspect we will want to continue trimming down modules and
dependencies if they don't have a clear purpose under the new objective of
a scalable activitypub backend plus useful plugins (such as the persistence
modules, data providers, and data augmentation modules in
streams-contrib). I think we could approach a form where our core
dependencies are apache projects only - commons, httpcomponents, juneau,
lucene/solr, maybe pekko? There will need to be at least one default data
repository capable of storing anything in the spec and supporting the
queries required by any endpoint. Jena seems like a good candidate for
this, at least for small-ish deployments, but there might be another better
solution I've not thought of yet.
P.S. It's nice to hear from everyone, I hope you are all doing well.
On Sun, Dec 17, 2023 at 9:10 PM Brian Hodge <br...@gmail.com> wrote:
> Apologies, this got buried in my inbox under other mailing lists.
>
> Steve, your suggestions make sense to me. I strongly agree that building on
> Streams to ship broadly useful software is a worthwhile goal.
>
> I do have some questions about the realignment you're envisioning. Have you
> written (or do you plan to write) anything more longform on design /
> architecture of the refactor you have in mind?
>
> I'm happy to help however I can. Are there near-term tasks that I and
> others can pitch in on?
>
> -Brian Hodge
>
> On Sun, Dec 17, 2023 at 2:58 PM Suneel Marthi <sm...@apache.org> wrote:
>
> > Replace ElasticSearch with OpenSearch ???
> >
> > On Sun, Dec 17, 2023 at 12:36 PM Steve Blackmon <sb...@apache.org>
> > wrote:
> >
> > > Thanks Trevor. Silent majority, hope to hear from you as/if the reboot
> > > picks up steam.
> > >
> > > I've gotten started the last few weeks by refactoring to enable builds
> > with
> > > Java 17 (committed) and a purge of vulnerable dependencies (almost
> > ready).
> > >
> > > It looks like most of of modules will be able to remain active in the
> > > reactor to achieve the above, possible exceptions of our hdfs and
> > > elasticsearch modules. Elasticsearch is very outdated and even the
> > newest
> > > available common-hdfs doesn't appear able to have the vulnerable
> > > commons-collections:commons-collections removed and still work.
> > >
> > > Stay tuned
> > >
> > > On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <trevor.d.grant@gmail.com
> >
> > > wrote:
> > >
> > > > I agree with most of what you're saying, candidly my bet is on social
> > > media
> > > > as we know it just sort of silently fading as Millenials get older.
> But
> > > I'm
> > > > still on board to help out- I have cycles to work tickets, but idk
> if I
> > > > have enough to put in any good faith effort to lead (we're doing a
> > > similar
> > > > reboot over on Mahout, where I am leading).
> > > >
> > > > tg
> > > >
> > > > On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <
> > > sblackmon@apache.org>
> > > > wrote:
> > > >
> > > > > Hello dev@streams.a.o,
> > > > >
> > > > > This list has been very quiet for a very long time. I’m hoping to
> > > change
> > > > > that, and generate some momentum toward a worth-while initiative
> > right
> > > > > in-line with Streams founding goals.
> > > > >
> > > > > The project founding proposal [1] includes the following goals:
> > > > >
> > > > > • Publication of Activities from multiple systems via HTTP
> > > > > • Aggregation and syndication of streams
> > > > > • Support for security trimming of streams by social graph
> > > > > • Noise reduction and intelligent filtering
> > > > > • Federation of streams across disparate systems
> > > > > • Provide libraries for easy integration in source systems
> > > > >
> > > > > The code we actually built focused most heavily on goals #2 and #5,
> > and
> > > > > evolved into a (useful IMO, at least around the time of our TLP
> > > > graduation)
> > > > > java SDK for building complex social media application back-end
> > > > > ’streams’. But, we never shipped software with clear stand-alone
> > > utility
> > > > > for anyone other than java developers. The internet, and the
> social
> > > > media
> > > > > landscape specifically, has changed a lot since then, and I would
> > argue
> > > > > that the system envisioned in the proposal is needed now more than
> > > > ever. I
> > > > > won’t go right now into all the reasons I believe that is the case,
> > > > because
> > > > > it would be TL;DR and off-topic, but will just say that there
> appears
> > > to
> > > > be
> > > > > an unprecedented (at least in this century) interest in hosting
> > > > > social-media-like software that can inter-operate across
> > installations.
> > > > >
> > > > > So I’m here asking whether anyone is open to a discussion about
> > > > > re-aligning this project to the totality of it’s origin, with a new
> > > North
> > > > > Star:
> > > > >
> > > > > • a containerized deployable software stack that provides an
> Activity
> > > > > Streams 2.0 / Activity Pub interface via rest API
> > > > > • Confirmed interoperable with the Mastodon front-end, and other
> > > > > ActivityPub client software
> > > > > • Built to scale with minimal system admin burden to the needs of
> the
> > > > > largest existing Mastodon instances: ballpark ~1M total users, 10K
> > > > > concurrent
> > > > > • Able to scale far beyond that for an experienced team: a
> > medium-sized
> > > > > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> > > > >
> > > > > I don’t think it’s hyper-bole to observe that the future of social
> > > media
> > > > > is in flux, and in early stages of a 10+ year trend toward
> > > > > decentralization. I for want really want that decentralized future
> > to
> > > be
> > > > > based on Activity Streams 2.0 and ActivityPub, now official W3C
> > > > standards,
> > > > > and thus inter-operable when desired, rather than completely
> silo’ed
> > > like
> > > > > the prior generation. Also I think the above is entirely plausible
> > on
> > > a
> > > > > 1-2 year time-scale.
> > > > >
> > > > > Does anyone else agree? Is anyone willing to support a pivot like
> > > this,
> > > > > and try to re-build this product and community with me?
> > > > >
> > > > > • [1]
> > > > >
> > https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> > > > >
> > > > >
> > > > > Steve Blackmon
> > > > > Apache Streams PMC Chair
> > > > >
> > > >
> > >
> >
>
Re: New Initiative
Posted by Brian Hodge <br...@gmail.com>.
Apologies, this got buried in my inbox under other mailing lists.
Steve, your suggestions make sense to me. I strongly agree that building on
Streams to ship broadly useful software is a worthwhile goal.
I do have some questions about the realignment you're envisioning. Have you
written (or do you plan to write) anything more longform on design /
architecture of the refactor you have in mind?
I'm happy to help however I can. Are there near-term tasks that I and
others can pitch in on?
-Brian Hodge
On Sun, Dec 17, 2023 at 2:58 PM Suneel Marthi <sm...@apache.org> wrote:
> Replace ElasticSearch with OpenSearch ???
>
> On Sun, Dec 17, 2023 at 12:36 PM Steve Blackmon <sb...@apache.org>
> wrote:
>
> > Thanks Trevor. Silent majority, hope to hear from you as/if the reboot
> > picks up steam.
> >
> > I've gotten started the last few weeks by refactoring to enable builds
> with
> > Java 17 (committed) and a purge of vulnerable dependencies (almost
> ready).
> >
> > It looks like most of of modules will be able to remain active in the
> > reactor to achieve the above, possible exceptions of our hdfs and
> > elasticsearch modules. Elasticsearch is very outdated and even the
> newest
> > available common-hdfs doesn't appear able to have the vulnerable
> > commons-collections:commons-collections removed and still work.
> >
> > Stay tuned
> >
> > On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <tr...@gmail.com>
> > wrote:
> >
> > > I agree with most of what you're saying, candidly my bet is on social
> > media
> > > as we know it just sort of silently fading as Millenials get older. But
> > I'm
> > > still on board to help out- I have cycles to work tickets, but idk if I
> > > have enough to put in any good faith effort to lead (we're doing a
> > similar
> > > reboot over on Mahout, where I am leading).
> > >
> > > tg
> > >
> > > On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <
> > sblackmon@apache.org>
> > > wrote:
> > >
> > > > Hello dev@streams.a.o,
> > > >
> > > > This list has been very quiet for a very long time. I’m hoping to
> > change
> > > > that, and generate some momentum toward a worth-while initiative
> right
> > > > in-line with Streams founding goals.
> > > >
> > > > The project founding proposal [1] includes the following goals:
> > > >
> > > > • Publication of Activities from multiple systems via HTTP
> > > > • Aggregation and syndication of streams
> > > > • Support for security trimming of streams by social graph
> > > > • Noise reduction and intelligent filtering
> > > > • Federation of streams across disparate systems
> > > > • Provide libraries for easy integration in source systems
> > > >
> > > > The code we actually built focused most heavily on goals #2 and #5,
> and
> > > > evolved into a (useful IMO, at least around the time of our TLP
> > > graduation)
> > > > java SDK for building complex social media application back-end
> > > > ’streams’. But, we never shipped software with clear stand-alone
> > utility
> > > > for anyone other than java developers. The internet, and the social
> > > media
> > > > landscape specifically, has changed a lot since then, and I would
> argue
> > > > that the system envisioned in the proposal is needed now more than
> > > ever. I
> > > > won’t go right now into all the reasons I believe that is the case,
> > > because
> > > > it would be TL;DR and off-topic, but will just say that there appears
> > to
> > > be
> > > > an unprecedented (at least in this century) interest in hosting
> > > > social-media-like software that can inter-operate across
> installations.
> > > >
> > > > So I’m here asking whether anyone is open to a discussion about
> > > > re-aligning this project to the totality of it’s origin, with a new
> > North
> > > > Star:
> > > >
> > > > • a containerized deployable software stack that provides an Activity
> > > > Streams 2.0 / Activity Pub interface via rest API
> > > > • Confirmed interoperable with the Mastodon front-end, and other
> > > > ActivityPub client software
> > > > • Built to scale with minimal system admin burden to the needs of the
> > > > largest existing Mastodon instances: ballpark ~1M total users, 10K
> > > > concurrent
> > > > • Able to scale far beyond that for an experienced team: a
> medium-sized
> > > > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> > > >
> > > > I don’t think it’s hyper-bole to observe that the future of social
> > media
> > > > is in flux, and in early stages of a 10+ year trend toward
> > > > decentralization. I for want really want that decentralized future
> to
> > be
> > > > based on Activity Streams 2.0 and ActivityPub, now official W3C
> > > standards,
> > > > and thus inter-operable when desired, rather than completely silo’ed
> > like
> > > > the prior generation. Also I think the above is entirely plausible
> on
> > a
> > > > 1-2 year time-scale.
> > > >
> > > > Does anyone else agree? Is anyone willing to support a pivot like
> > this,
> > > > and try to re-build this product and community with me?
> > > >
> > > > • [1]
> > > >
> https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> > > >
> > > >
> > > > Steve Blackmon
> > > > Apache Streams PMC Chair
> > > >
> > >
> >
>
Re: New Initiative
Posted by Suneel Marthi <sm...@apache.org>.
Replace ElasticSearch with OpenSearch ???
On Sun, Dec 17, 2023 at 12:36 PM Steve Blackmon <sb...@apache.org>
wrote:
> Thanks Trevor. Silent majority, hope to hear from you as/if the reboot
> picks up steam.
>
> I've gotten started the last few weeks by refactoring to enable builds with
> Java 17 (committed) and a purge of vulnerable dependencies (almost ready).
>
> It looks like most of of modules will be able to remain active in the
> reactor to achieve the above, possible exceptions of our hdfs and
> elasticsearch modules. Elasticsearch is very outdated and even the newest
> available common-hdfs doesn't appear able to have the vulnerable
> commons-collections:commons-collections removed and still work.
>
> Stay tuned
>
> On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <tr...@gmail.com>
> wrote:
>
> > I agree with most of what you're saying, candidly my bet is on social
> media
> > as we know it just sort of silently fading as Millenials get older. But
> I'm
> > still on board to help out- I have cycles to work tickets, but idk if I
> > have enough to put in any good faith effort to lead (we're doing a
> similar
> > reboot over on Mahout, where I am leading).
> >
> > tg
> >
> > On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <
> sblackmon@apache.org>
> > wrote:
> >
> > > Hello dev@streams.a.o,
> > >
> > > This list has been very quiet for a very long time. I’m hoping to
> change
> > > that, and generate some momentum toward a worth-while initiative right
> > > in-line with Streams founding goals.
> > >
> > > The project founding proposal [1] includes the following goals:
> > >
> > > • Publication of Activities from multiple systems via HTTP
> > > • Aggregation and syndication of streams
> > > • Support for security trimming of streams by social graph
> > > • Noise reduction and intelligent filtering
> > > • Federation of streams across disparate systems
> > > • Provide libraries for easy integration in source systems
> > >
> > > The code we actually built focused most heavily on goals #2 and #5, and
> > > evolved into a (useful IMO, at least around the time of our TLP
> > graduation)
> > > java SDK for building complex social media application back-end
> > > ’streams’. But, we never shipped software with clear stand-alone
> utility
> > > for anyone other than java developers. The internet, and the social
> > media
> > > landscape specifically, has changed a lot since then, and I would argue
> > > that the system envisioned in the proposal is needed now more than
> > ever. I
> > > won’t go right now into all the reasons I believe that is the case,
> > because
> > > it would be TL;DR and off-topic, but will just say that there appears
> to
> > be
> > > an unprecedented (at least in this century) interest in hosting
> > > social-media-like software that can inter-operate across installations.
> > >
> > > So I’m here asking whether anyone is open to a discussion about
> > > re-aligning this project to the totality of it’s origin, with a new
> North
> > > Star:
> > >
> > > • a containerized deployable software stack that provides an Activity
> > > Streams 2.0 / Activity Pub interface via rest API
> > > • Confirmed interoperable with the Mastodon front-end, and other
> > > ActivityPub client software
> > > • Built to scale with minimal system admin burden to the needs of the
> > > largest existing Mastodon instances: ballpark ~1M total users, 10K
> > > concurrent
> > > • Able to scale far beyond that for an experienced team: a medium-sized
> > > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> > >
> > > I don’t think it’s hyper-bole to observe that the future of social
> media
> > > is in flux, and in early stages of a 10+ year trend toward
> > > decentralization. I for want really want that decentralized future to
> be
> > > based on Activity Streams 2.0 and ActivityPub, now official W3C
> > standards,
> > > and thus inter-operable when desired, rather than completely silo’ed
> like
> > > the prior generation. Also I think the above is entirely plausible on
> a
> > > 1-2 year time-scale.
> > >
> > > Does anyone else agree? Is anyone willing to support a pivot like
> this,
> > > and try to re-build this product and community with me?
> > >
> > > • [1]
> > > https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> > >
> > >
> > > Steve Blackmon
> > > Apache Streams PMC Chair
> > >
> >
>
Re: New Initiative
Posted by Steve Blackmon <sb...@apache.org>.
Thanks Trevor. Silent majority, hope to hear from you as/if the reboot
picks up steam.
I've gotten started the last few weeks by refactoring to enable builds with
Java 17 (committed) and a purge of vulnerable dependencies (almost ready).
It looks like most of of modules will be able to remain active in the
reactor to achieve the above, possible exceptions of our hdfs and
elasticsearch modules. Elasticsearch is very outdated and even the newest
available common-hdfs doesn't appear able to have the vulnerable
commons-collections:commons-collections removed and still work.
Stay tuned
On Mon, Dec 4, 2023 at 10:11 AM Trevor Grant <tr...@gmail.com>
wrote:
> I agree with most of what you're saying, candidly my bet is on social media
> as we know it just sort of silently fading as Millenials get older. But I'm
> still on board to help out- I have cycles to work tickets, but idk if I
> have enough to put in any good faith effort to lead (we're doing a similar
> reboot over on Mahout, where I am leading).
>
> tg
>
> On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <sb...@apache.org>
> wrote:
>
> > Hello dev@streams.a.o,
> >
> > This list has been very quiet for a very long time. I’m hoping to change
> > that, and generate some momentum toward a worth-while initiative right
> > in-line with Streams founding goals.
> >
> > The project founding proposal [1] includes the following goals:
> >
> > • Publication of Activities from multiple systems via HTTP
> > • Aggregation and syndication of streams
> > • Support for security trimming of streams by social graph
> > • Noise reduction and intelligent filtering
> > • Federation of streams across disparate systems
> > • Provide libraries for easy integration in source systems
> >
> > The code we actually built focused most heavily on goals #2 and #5, and
> > evolved into a (useful IMO, at least around the time of our TLP
> graduation)
> > java SDK for building complex social media application back-end
> > ’streams’. But, we never shipped software with clear stand-alone utility
> > for anyone other than java developers. The internet, and the social
> media
> > landscape specifically, has changed a lot since then, and I would argue
> > that the system envisioned in the proposal is needed now more than
> ever. I
> > won’t go right now into all the reasons I believe that is the case,
> because
> > it would be TL;DR and off-topic, but will just say that there appears to
> be
> > an unprecedented (at least in this century) interest in hosting
> > social-media-like software that can inter-operate across installations.
> >
> > So I’m here asking whether anyone is open to a discussion about
> > re-aligning this project to the totality of it’s origin, with a new North
> > Star:
> >
> > • a containerized deployable software stack that provides an Activity
> > Streams 2.0 / Activity Pub interface via rest API
> > • Confirmed interoperable with the Mastodon front-end, and other
> > ActivityPub client software
> > • Built to scale with minimal system admin burden to the needs of the
> > largest existing Mastodon instances: ballpark ~1M total users, 10K
> > concurrent
> > • Able to scale far beyond that for an experienced team: a medium-sized
> > Web 2.0 social network: ~100M total users, up to ~1M concurrent
> >
> > I don’t think it’s hyper-bole to observe that the future of social media
> > is in flux, and in early stages of a 10+ year trend toward
> > decentralization. I for want really want that decentralized future to be
> > based on Activity Streams 2.0 and ActivityPub, now official W3C
> standards,
> > and thus inter-operable when desired, rather than completely silo’ed like
> > the prior generation. Also I think the above is entirely plausible on a
> > 1-2 year time-scale.
> >
> > Does anyone else agree? Is anyone willing to support a pivot like this,
> > and try to re-build this product and community with me?
> >
> > • [1]
> > https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
> >
> >
> > Steve Blackmon
> > Apache Streams PMC Chair
> >
>
Re: New Initiative
Posted by Trevor Grant <tr...@gmail.com>.
I agree with most of what you're saying, candidly my bet is on social media
as we know it just sort of silently fading as Millenials get older. But I'm
still on board to help out- I have cycles to work tickets, but idk if I
have enough to put in any good faith effort to lead (we're doing a similar
reboot over on Mahout, where I am leading).
tg
On Sun, Dec 3, 2023 at 8:21 PM sblackmon@apache.org <sb...@apache.org>
wrote:
> Hello dev@streams.a.o,
>
> This list has been very quiet for a very long time. I’m hoping to change
> that, and generate some momentum toward a worth-while initiative right
> in-line with Streams founding goals.
>
> The project founding proposal [1] includes the following goals:
>
> • Publication of Activities from multiple systems via HTTP
> • Aggregation and syndication of streams
> • Support for security trimming of streams by social graph
> • Noise reduction and intelligent filtering
> • Federation of streams across disparate systems
> • Provide libraries for easy integration in source systems
>
> The code we actually built focused most heavily on goals #2 and #5, and
> evolved into a (useful IMO, at least around the time of our TLP graduation)
> java SDK for building complex social media application back-end
> ’streams’. But, we never shipped software with clear stand-alone utility
> for anyone other than java developers. The internet, and the social media
> landscape specifically, has changed a lot since then, and I would argue
> that the system envisioned in the proposal is needed now more than ever. I
> won’t go right now into all the reasons I believe that is the case, because
> it would be TL;DR and off-topic, but will just say that there appears to be
> an unprecedented (at least in this century) interest in hosting
> social-media-like software that can inter-operate across installations.
>
> So I’m here asking whether anyone is open to a discussion about
> re-aligning this project to the totality of it’s origin, with a new North
> Star:
>
> • a containerized deployable software stack that provides an Activity
> Streams 2.0 / Activity Pub interface via rest API
> • Confirmed interoperable with the Mastodon front-end, and other
> ActivityPub client software
> • Built to scale with minimal system admin burden to the needs of the
> largest existing Mastodon instances: ballpark ~1M total users, 10K
> concurrent
> • Able to scale far beyond that for an experienced team: a medium-sized
> Web 2.0 social network: ~100M total users, up to ~1M concurrent
>
> I don’t think it’s hyper-bole to observe that the future of social media
> is in flux, and in early stages of a 10+ year trend toward
> decentralization. I for want really want that decentralized future to be
> based on Activity Streams 2.0 and ActivityPub, now official W3C standards,
> and thus inter-operable when desired, rather than completely silo’ed like
> the prior generation. Also I think the above is entirely plausible on a
> 1-2 year time-scale.
>
> Does anyone else agree? Is anyone willing to support a pivot like this,
> and try to re-build this product and community with me?
>
> • [1]
> https://cwiki.apache.org/confluence/display/incubator/StreamsProposal
>
>
> Steve Blackmon
> Apache Streams PMC Chair
>