[DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

Hi Everyone,

I’ve created a draft PR (https://github.com/apache/airflow/pull/46977) to
introduce and discuss a new provider for using Gremlin—the graph traversal
language of Apache TinkerPop (more details here:
https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by various
graph database vendors such as Azure Cosmos DB and Amazon Neptune.
Previously, I had to develop a custom hook to query data from Azure Cosmos
DB using Apache Gremlin.

I managed to create a provider and run it locally on the main branch.
However, I ran into the BaseHook issue (
https://github.com/apache/airflow/issues/45233) on that branch, so I ended
up testing it fully on the v2-10-test branch. The PR should be complete,
but I’ve kept it as a draft for now while we discuss the provider.

I’m a new contributor, so I’m especially eager to hear your feedback.
Comments on the PR is very welcome, and please feel free to reach out with
any questions via email or Slack.

Thanks,
Ahmad Farhan

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

Hey Stephen!

I am still waiting for review from code maintainers and they are currently
busy with Airflow 3. Also, recently there had been major code changes
(after uv PR) that are raising a lot of errors so some work is needed on my
side which is WIP.

Thanks,
Farhan

On Fri, 4 Apr 2025 at 1:47 pm, Stephen Mallette <sp...@gmail.com>
wrote:

> Hi everyone, I just wanted to check in on the status of the TinkerPop
> Provider PR:
>
> https://github.com/apache/airflow/pull/47446
>
> I don't know if there is any outstanding work left here but I was curious
> if there was any sense from committers as to when this body of work will be
> merged and made available in a release?
>
> Thanks,
>
> Stephen
>
>
> On Sat, Mar 8, 2025 at 2:57 PM Paul King <pa...@apache.org> wrote:
>
> >
> > Nice to see the Apache TinkerPop/Gremlin support. As well as using
> Gremlin
> > with TinkerPop's TinkerGraph, I have also used it with OrientDB,
> ArcadeDB,
> > and Apache HugeGraph. If anyone is interested, I did a blog post here:
> > https://groovy.apache.org/blog/groovy-graph-databases
> > (Sorry but it's Groovy not Python)
> >
> > Cheers, Paul.
> >
> > On 2025/03/08 11:34:11 Ahmad Farhan wrote:
> > > Hi!
> > > I managed to do the last cleanup on the PR, and is ready for thorough
> > > review. I will create a  `[LAZY CONSENSUS]` thread on Monday/Tuesday.
> > Feel
> > > free to review and comment on the PR in the meantime.
> > >
> > > Farhan
> > >
> > > On Thu, Mar 6, 2025 at 1:40 PM Stephen Mallette <sp...@gmail.com>
> > > wrote:
> > >
> > > > Hi Ahmad, thanks for the updates! I've spent some time looking at the
> > > > changes and have added some comments and questions. I've pointed out
> a
> > > > couple things for future work that I think will be important, but
> > nothing
> > > > that needs to be changed for this PR in my mind. Looking forward to
> > seeing
> > > > your responses, other community feedback, and ultimately a merge of
> the
> > > > provider - take care!
> > > >
> > > > On Thu, Mar 6, 2025 at 7:28 AM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello again!
> > > > > After several days of debugging and branch restructuring I managed
> to
> > > > > create another PR https://github.com/apache/airflow/pull/47446,
> > this is
> > > > > due
> > > > > to major merge conflicts after the www clean up.
> > > > > The renaming of the provider to apache-tinkerpop is done but I kept
> > the
> > > > > integration testing as 'gremlin' to avoid the conflict/confusion
> > with the
> > > > > server name within the CI.
> > > > >
> > > > > Please review the PR thoroughly and I am happy to work on the
> > following
> > > > > steps.
> > > > >
> > > > > Thanks,
> > > > > Farhan
> > > > >
> > > > > On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I did look into the naming and I thought that it would need to be
> > > > > > discussed at some point before the lazy consensus stage after the
> > dev
> > > > > work
> > > > > > is done but I guess I was wrong :)
> > > > > >
> > > > > > I read through some docs regarding the naming and I kept thinking
> > that
> > > > > > Apache Gremlin might not be right, so I decided to remove
> 'Apache'
> > from
> > > > > all
> > > > > > doc strings. One thing that popped in one of the documentations
> > from
> > > > > > Microsoft that says Apache Gremlin (here
> > > > > >
> > https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction
> > > > )
> > > > > > which confused me.
> > > > > >
> > > > > > I will change the folder to apache/tinkerpop. Thanks Stephen for
> > the
> > > > > > in-depth explanation.
> > > > > >
> > > > > > On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > > >
> > > > > >> Cool. I will let Ahmad comment, but I think we found the
> > **someone**
> > > > who
> > > > > >> will help in case there are some future issues with the
> > > > > Tinkerpop/Gremlin
> > > > > >> provider.
> > > > > >> While I like Gremlin better (it's just a cool name and I like
> the
> > > > logo,
> > > > > >> tinkerpop has a cool logo as well
> > > > > https://tinkerpop.apache.org/index.html
> > > > > >> ).
> > > > > >>
> > > > > >> So as long as we decide not to use common.graph -> I am fine
> with
> > both
> > > > > :)
> > > > > >>
> > > > > >> j.
> > > > > >>
> > > > > >>
> > > > > >> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <
> > > > spmallette@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <
> jarek@potiuk.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > > In the interest of ASF trademarks, I would suggest it be
> > called
> > > > > >> > > "apache/tinkerpop" with "Gremlin" naming reserved for
> > operators
> > > > and
> > > > > >> the
> > > > > >> > > like, as it is now with GremlinOperator. I think this makes
> > sense
> > > > > >> because
> > > > > >> > > it is connecting to TinkerPop-enabled systems via Gremlin. I
> > would
> > > > > >> > > similarly suggest that references to "Apache Gremlin" and
> the
> > like
> > > > > >> become
> > > > > >> > > "Apache TinkerPop".
> > > > > >> > >
> > > > > >> > > That's an interesting one - indeed TinkerPop is the PMC/
> > > > Framework -
> > > > > >> > > Gremlin is the language.
> > > > > >> > >
> > > > > >> > > I am not sure we are actually using TinkerPop here - because
> > > > > >> TinkerPop is
> > > > > >> > > the whole framework - Ahmad, can you explain the relation
> > there -
> > > > > are
> > > > > >> > those
> > > > > >> > > other systems simply implement Gremlin as language or do
> they
> > use
> > > > > >> > TinkerPop
> > > > > >> > > for something / as a backend?
> > > > > >> > >
> > > > > >> >
> > > > > >> > I'm sure Ahmad could answer but I'll quickly offer my take. I
> > think
> > > > > >> that in
> > > > > >> > this case we should prefer "TinkerPop" over Gremlin as a
> > top-level
> > > > > name
> > > > > >> > particularly because it's prefixed with "Apache" and there is
> no
> > > > > "Apache
> > > > > >> > Gremlin" which I tend to think is confusing when the words are
> > that
> > > > > >> close
> > > > > >> > together. I can't recall over the years just how many times
> I've
> > > > asked
> > > > > >> for
> > > > > >> > corrections in blog posts. :)
> > > > > >> >
> > > > > >> > Because that's a bit of a conceptual difference here. For
> > example in
> > > > > the
> > > > > >> > > provider we are importing
> > https://pypi.org/project/gremlinpython
> > > > > not
> > > > > >> > > "tinkerpop" - and it also does not have tinkerpop as
> > dependency.
> > > > > >> > >
> > > > > >> >
> > > > > >> > A bit of history goes along with a lot of our naming for what
> we
> > > > term
> > > > > >> > Gremlin Language Variants (GLVs), like gremlinpython, which
> are
> > > > > >> variants of
> > > > > >> > Gremlin natively implemented to allow users to express Gremlin
> > in
> > > > the
> > > > > >> > idioms of their own language. They also provide driver
> > connectivity
> > > > to
> > > > > >> > compatible servers. TinkerPop has mostly inherited all of its
> > > > language
> > > > > >> > variants, including gremlinpython which was the first, from
> > > > > third-party
> > > > > >> > community developers. As a project, we didn't really get a
> hand
> > in
> > > > the
> > > > > >> > naming so with those projects already in heavy use we just
> > kinda of
> > > > > >> stuck
> > > > > >> > to it and even doubled-down (like when we built gremlin-go
> > within
> > > > the
> > > > > >> ASF).
> > > > > >> >
> > > > > >> > I think in this case, your project organization under "apache"
> > seems
> > > > > to
> > > > > >> > almost lend itself nicely to apache/tinkerpop. i think users
> > will
> > > > > >> recognize
> > > > > >> > it as equally as they recognize Gremlin.
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we
> > > > should
> > > > > >> ask
> > > > > >> > > tinkerpop PMC what they think about it?
> > > > > >> > >
> > > > > >> >
> > > > > >> > Gremlin is not an ASF trademark. That was debated for quite a
> > long
> > > > > time
> > > > > >> > with trademarks@ along with deciding if Gremlin, the
> character
> > and
> > > > > his
> > > > > >> > friends[1], were to be protected. In the end, for reasons I'm
> > not
> > > > > sure I
> > > > > >> > quite remember, the ASF didn't think it was necessary.
> > > > > >> >
> > > > > >> > Anyway, I'm not sure if you noted my earlier post[2] but I'm
> > one of
> > > > > the
> > > > > >> > original contributors to Apache TinkerPop, even before we
> > brought it
> > > > > to
> > > > > >> the
> > > > > >> > ASF so I'm pretty familiar with our project.  :)
> > > > > >> >
> > > > > >> > [1]
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> > > > > >> > [2]
> > > > https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> > > > > >> >
> > > > > >> >
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > J.
> > > > > >> > >
> > > > > >> > >
> > > > > >> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> > > > > >> spmallette@apache.org>
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > > > >> > > > > Yeah . `apache/gremlin" seems like a better option then.
> > Does
> > > > > >> anyone
> > > > > >> > > have
> > > > > >> > > > > anything against it?
> > > > > >> > > >
> > > > > >> > > > In the interest of ASF trademarks, I would suggest it be
> > called
> > > > > >> > > > "apache/tinkerpop" with "Gremlin" naming reserved for
> > operators
> > > > > and
> > > > > >> the
> > > > > >> > > > like, as it is now with GremlinOperator. I think this
> makes
> > > > sense
> > > > > >> > because
> > > > > >> > > > it is connecting to TinkerPop-enabled systems via
> Gremlin. I
> > > > would
> > > > > >> > > > similarly suggest that references to "Apache Gremlin" and
> > the
> > > > like
> > > > > >> > become
> > > > > >> > > > "Apache TinkerPop".
> > > > > >> > > >
> > > > > >> > > > > I think we are pretty happy with accepting "other
> > > > > >> > > > > apache" projects as providers, so I see no issue with
> > Gremlin
> > > > -
> > > > > >> > knowing
> > > > > >> > > > > that we can always reach out to our friendly Apache
> > Community
> > > > in
> > > > > >> case
> > > > > >> > > of
> > > > > >> > > > > any issues. So - unless we do not hear any "opposition"
> > in a
> > > > few
> > > > > >> > days,
> > > > > >> > > I
> > > > > >> > > > > think it would make sense if you start `[LAZY
> CONSENSUS]`
> > > > > thread -
> > > > > >> > > > > without a need for `[VOTE]` thread.
> > > > > >> > > > >
> > > > > >> > > > > One thing though that I would love to have - is to also
> > have
> > > > an
> > > > > >> > > > integration
> > > > > >> > > > > test if possible (we had it with apache.kafka for
> > example) -
> > > > > those
> > > > > >> > are
> > > > > >> > > > > tests that could run **some** graphdb database locally
> > (via
> > > > > >> > > > docker-compose)
> > > > > >> > > > > and run a very rudimentary checks against a "real"
> > database,
> > > > > not a
> > > > > >> > > mocked
> > > > > >> > > > > call. That would make it more robust.
> > > > > >> > > > >
> > > > > >> > > > > More about integration tests, how to build, run, test
> > them and
> > > > > >> > > integrate
> > > > > >> > > > > them in our CI can be found here:
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > > > >> > > > > - happy to help if you are stuck with it.
> > > > > >> > > > >
> > > > > >> > > > > J.
> > > > > >> > > > >
> > > > > >> > > > >
> > > > > >> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > > > > >> > > ahmad.farhan9715@gmail.com
> > > > > >> > > > >
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > I pushed changes to move the provider into the
> “apache”
> > > > > >> directory.
> > > > > >> > > > After
> > > > > >> > > > > > updating the class references across the project, I
> > > > re-tested
> > > > > >> and
> > > > > >> > all
> > > > > >> > > > tests
> > > > > >> > > > > > passed.
> > > > > >> > > > > >
> > > > > >> > > > > > Regarding the use of Gremlin (or another graph query
> > > > language
> > > > > >> like
> > > > > >> > > > Cypher
> > > > > >> > > > > > and SPARQL) for a common package approach, here are my
> > > > > thoughts
> > > > > >> on
> > > > > >> > > the
> > > > > >> > > > pros
> > > > > >> > > > > > and cons:
> > > > > >> > > > > >
> > > > > >> > > > > > pros (I can see only one):
> > > > > >> > > > > >
> > > > > >> > > > > >    - Gremlin has been widely adopted by different
> cloud
> > > > > vendors
> > > > > >> > (e.g.
> > > > > >> > > > Azure
> > > > > >> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as
> > well as
> > > > > in
> > > > > >> > > > self-hosted
> > > > > >> > > > > >    environments.
> > > > > >> > > > > >
> > > > > >> > > > > > cons:
> > > > > >> > > > > >
> > > > > >> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL
> each
> > have
> > > > > >> their
> > > > > >> > > own
> > > > > >> > > > > >    drivers for executing queries.
> > > > > >> > > > > >    - To achieve a common abstraction, a wrapper around
> > each
> > > > > >> driver
> > > > > >> > > > would be
> > > > > >> > > > > >    required. Each driver has its own connection
> > parameters,
> > > > > >> > > underlying
> > > > > >> > > > > >    protocols, and may need method overrides for
> > > > compatibility
> > > > > >> with
> > > > > >> > > > > > different
> > > > > >> > > > > >    Python versions.
> > > > > >> > > > > >    - Not all vendors support every query language; for
> > > > > instance,
> > > > > >> > > > Gremlin
> > > > > >> > > > > >    for Neo4j has been deprecated in recent releases,
> > while
> > > > > >> Cosmos
> > > > > >> > DB
> > > > > >> > > > does
> > > > > >> > > > > > not
> > > > > >> > > > > >    support Cypher or SPARQL.
> > > > > >> > > > > >
> > > > > >> > > > > > While it would be ideal to have a unified graph query
> > > > language
> > > > > >> and
> > > > > >> > > > driver
> > > > > >> > > > > > that works seamlessly across different vendors, such a
> > > > > solution
> > > > > >> > does
> > > > > >> > > > not
> > > > > >> > > > > > exist at the moment. In my opinion, implementing
> > > > > >> provider-specific
> > > > > >> > > > > > solutions for each query language (Gremlin, Cypher,
> > SPARQL)
> > > > is
> > > > > >> more
> > > > > >> > > > > > realistic and practical given the current landscape.
> > > > > >> > > > > >
> > > > > >> > > > > > Happy to discuss further or answer any questions!
> > > > > >> > > > > >
> > > > > >> > > > > > Farhan
> > > > > >> > > > > >
> > > > > >> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > > > > >> > > > ahmad.farhan9715@gmail.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > I have worked with two different graph database
> > > > > vendors—Azure
> > > > > >> > > Cosmos
> > > > > >> > > > DB
> > > > > >> > > > > > > and Neo4j. During our migration to Neo4j, we
> > discovered
> > > > that
> > > > > >> > using
> > > > > >> > > > the
> > > > > >> > > > > > > Gremlin language wasn’t possible; we were forced to
> > > > rewrite
> > > > > >> all
> > > > > >> > our
> > > > > >> > > > > > queries
> > > > > >> > > > > > > into Cypher, which is the native language for Neo4j
> > and,
> > > > in
> > > > > my
> > > > > >> > > > > > experience,
> > > > > >> > > > > > > much simpler for querying.
> > > > > >> > > > > > >
> > > > > >> > > > > > > This situation highlights a key challenge for a
> common
> > > > > >> > abstraction:
> > > > > >> > > > the
> > > > > >> > > > > > > underlying query languages and
> > connection/authentication
> > > > > >> > mechanisms
> > > > > >> > > > vary
> > > > > >> > > > > > > significantly. Gremlin is not only different from
> > Cypher
> > > > in
> > > > > >> > syntax
> > > > > >> > > > but is
> > > > > >> > > > > > > also deprecated for Neo4j (see
> > > > > >> > > > > > >
> > > > > >>
> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> > > > > >> > ).
> > > > > >> > > > > > >
> > > > > >> > > > > > > The question would be how can the common approach
> > > > > accommodate
> > > > > >> > these
> > > > > >> > > > > > > different query languages?
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
> > > > > >> jarek@potiuk.com>
> > > > > >> > > > wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > >> Without deep looking at the code I love the idea -
> > it's
> > > > > very
> > > > > >> > > > similar to
> > > > > >> > > > > > >> what we have for common.sql and common.io - and
> soon
> > > > > >> > > > common.messaging
> > > > > >> > > > > > - I
> > > > > >> > > > > > >> also - long time ago - suggested common.dataframe
> > that
> > > > > >> someone
> > > > > >> > > could
> > > > > >> > > > > > >> submit
> > > > > >> > > > > > >> using Apache Ibis:
> > > > > >> > > > > > >>
> > > > > >> >
> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > > > > >> > > -
> > > > > >> > > > > > >> similarly I believe there was an idea about
> > common.llm
> > > > ...
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> I think the "common" pattern is a great one for
> > Airflow,
> > > > to
> > > > > >> > build
> > > > > >> > > > on top
> > > > > >> > > > > > >> of
> > > > > >> > > > > > >> "other giants" who build those common abstractions
> > that
> > > > you
> > > > > >> can
> > > > > >> > > > easily
> > > > > >> > > > > > >> switch between different implementations of various
> > data
> > > > > >> access
> > > > > >> > > > layers.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> My suggestion and question - would be however (not
> > very
> > > > > >> strong
> > > > > >> > on
> > > > > >> > > > it, I
> > > > > >> > > > > > >> would love to hear what others think, I know it's
> > been
> > > > > >> somewhat
> > > > > >> > > > > > >> contentious
> > > > > >> > > > > > >> when I started the ibis discussion) - would be to
> > make it
> > > > > >> > > > > > "common.graph",
> > > > > >> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> > > > > >> > "apache.ibis"
> > > > > >> > > -
> > > > > >> > > > just
> > > > > >> > > > > > >> to
> > > > > >> > > > > > >> stress that those are not implementations of
> > particular
> > > > > >> service
> > > > > >> > > but
> > > > > >> > > > > > >> opinionated choice of particular technology to do
> > > > "common"
> > > > > >> > > > operations.
> > > > > >> > > > > > >> This
> > > > > >> > > > > > >> is what essentially "common.io" is . - it should
> be
> > > > named
> > > > > >> > > "fsspec"
> > > > > >> > > > > > >> provider
> > > > > >> > > > > > >> if we were to name it by the "library" that
> > implemented
> > > > it.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> J.
> > > > > >> > > > > > >>
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > > > >> > > > > > ahmad.farhan9715@gmail.com>
> > > > > >> > > > > > >> wrote:
> > > > > >> > > > > > >>
> > > > > >> > > > > > >> > Hi Everyone,
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > I’ve created a draft PR (
> > > > > >> > > > https://github.com/apache/airflow/pull/46977
> > > > > >> > > > > > )
> > > > > >> > > > > > >> to
> > > > > >> > > > > > >> > introduce and discuss a new provider for using
> > > > > Gremlin—the
> > > > > >> > graph
> > > > > >> > > > > > >> traversal
> > > > > >> > > > > > >> > language of Apache TinkerPop (more details here:
> > > > > >> > > > > > >> > https://tinkerpop.apache.org/gremlin.html).
> > Gremlin is
> > > > > >> > > supported
> > > > > >> > > > by
> > > > > >> > > > > > >> > various
> > > > > >> > > > > > >> > graph database vendors such as Azure Cosmos DB
> and
> > > > Amazon
> > > > > >> > > Neptune.
> > > > > >> > > > > > >> > Previously, I had to develop a custom hook to
> query
> > > > data
> > > > > >> from
> > > > > >> > > > Azure
> > > > > >> > > > > > >> Cosmos
> > > > > >> > > > > > >> > DB using Apache Gremlin.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > I managed to create a provider and run it locally
> > on
> > > > the
> > > > > >> main
> > > > > >> > > > branch.
> > > > > >> > > > > > >> > However, I ran into the BaseHook issue (
> > > > > >> > > > > > >> > https://github.com/apache/airflow/issues/45233)
> on
> > > > that
> > > > > >> > branch,
> > > > > >> > > > so I
> > > > > >> > > > > > >> ended
> > > > > >> > > > > > >> > up testing it fully on the v2-10-test branch. The
> > PR
> > > > > >> should be
> > > > > >> > > > > > complete,
> > > > > >> > > > > > >> > but I’ve kept it as a draft for now while we
> > discuss
> > > > the
> > > > > >> > > provider.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > I’m a new contributor, so I’m especially eager to
> > hear
> > > > > your
> > > > > >> > > > feedback.
> > > > > >> > > > > > >> > Comments on the PR is very welcome, and please
> feel
> > > > free
> > > > > to
> > > > > >> > > reach
> > > > > >> > > > out
> > > > > >> > > > > > >> with
> > > > > >> > > > > > >> > any questions via email or Slack.
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >> > Thanks,
> > > > > >> > > > > > >> > Ahmad Farhan
> > > > > >> > > > > > >> >
> > > > > >> > > > > > >>
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > > >
> > > > > >>
> > ---------------------------------------------------------------------
> > > > > >> > > > To unsubscribe, e-mail:
> dev-unsubscribe@airflow.apache.org
> > > > > >> > > > For additional commands, e-mail:
> > dev-help@airflow.apache.org
> > > > > >> > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Stephen Mallette <sp...@gmail.com>]

Hi everyone, I just wanted to check in on the status of the TinkerPop
Provider PR:

https://github.com/apache/airflow/pull/47446

I don't know if there is any outstanding work left here but I was curious
if there was any sense from committers as to when this body of work will be
merged and made available in a release?

Thanks,

Stephen


On Sat, Mar 8, 2025 at 2:57 PM Paul King <pa...@apache.org> wrote:

>
> Nice to see the Apache TinkerPop/Gremlin support. As well as using Gremlin
> with TinkerPop's TinkerGraph, I have also used it with OrientDB, ArcadeDB,
> and Apache HugeGraph. If anyone is interested, I did a blog post here:
> https://groovy.apache.org/blog/groovy-graph-databases
> (Sorry but it's Groovy not Python)
>
> Cheers, Paul.
>
> On 2025/03/08 11:34:11 Ahmad Farhan wrote:
> > Hi!
> > I managed to do the last cleanup on the PR, and is ready for thorough
> > review. I will create a  `[LAZY CONSENSUS]` thread on Monday/Tuesday.
> Feel
> > free to review and comment on the PR in the meantime.
> >
> > Farhan
> >
> > On Thu, Mar 6, 2025 at 1:40 PM Stephen Mallette <sp...@gmail.com>
> > wrote:
> >
> > > Hi Ahmad, thanks for the updates! I've spent some time looking at the
> > > changes and have added some comments and questions. I've pointed out a
> > > couple things for future work that I think will be important, but
> nothing
> > > that needs to be changed for this PR in my mind. Looking forward to
> seeing
> > > your responses, other community feedback, and ultimately a merge of the
> > > provider - take care!
> > >
> > > On Thu, Mar 6, 2025 at 7:28 AM Ahmad Farhan <
> ahmad.farhan9715@gmail.com>
> > > wrote:
> > >
> > > > Hello again!
> > > > After several days of debugging and branch restructuring I managed to
> > > > create another PR https://github.com/apache/airflow/pull/47446,
> this is
> > > > due
> > > > to major merge conflicts after the www clean up.
> > > > The renaming of the provider to apache-tinkerpop is done but I kept
> the
> > > > integration testing as 'gremlin' to avoid the conflict/confusion
> with the
> > > > server name within the CI.
> > > >
> > > > Please review the PR thoroughly and I am happy to work on the
> following
> > > > steps.
> > > >
> > > > Thanks,
> > > > Farhan
> > > >
> > > > On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <
> ahmad.farhan9715@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I did look into the naming and I thought that it would need to be
> > > > > discussed at some point before the lazy consensus stage after the
> dev
> > > > work
> > > > > is done but I guess I was wrong :)
> > > > >
> > > > > I read through some docs regarding the naming and I kept thinking
> that
> > > > > Apache Gremlin might not be right, so I decided to remove 'Apache'
> from
> > > > all
> > > > > doc strings. One thing that popped in one of the documentations
> from
> > > > > Microsoft that says Apache Gremlin (here
> > > > >
> https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction
> > > )
> > > > > which confused me.
> > > > >
> > > > > I will change the folder to apache/tinkerpop. Thanks Stephen for
> the
> > > > > in-depth explanation.
> > > > >
> > > > > On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > > >
> > > > >> Cool. I will let Ahmad comment, but I think we found the
> **someone**
> > > who
> > > > >> will help in case there are some future issues with the
> > > > Tinkerpop/Gremlin
> > > > >> provider.
> > > > >> While I like Gremlin better (it's just a cool name and I like the
> > > logo,
> > > > >> tinkerpop has a cool logo as well
> > > > https://tinkerpop.apache.org/index.html
> > > > >> ).
> > > > >>
> > > > >> So as long as we decide not to use common.graph -> I am fine with
> both
> > > > :)
> > > > >>
> > > > >> j.
> > > > >>
> > > > >>
> > > > >> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <
> > > spmallette@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > > In the interest of ASF trademarks, I would suggest it be
> called
> > > > >> > > "apache/tinkerpop" with "Gremlin" naming reserved for
> operators
> > > and
> > > > >> the
> > > > >> > > like, as it is now with GremlinOperator. I think this makes
> sense
> > > > >> because
> > > > >> > > it is connecting to TinkerPop-enabled systems via Gremlin. I
> would
> > > > >> > > similarly suggest that references to "Apache Gremlin" and the
> like
> > > > >> become
> > > > >> > > "Apache TinkerPop".
> > > > >> > >
> > > > >> > > That's an interesting one - indeed TinkerPop is the PMC/
> > > Framework -
> > > > >> > > Gremlin is the language.
> > > > >> > >
> > > > >> > > I am not sure we are actually using TinkerPop here - because
> > > > >> TinkerPop is
> > > > >> > > the whole framework - Ahmad, can you explain the relation
> there -
> > > > are
> > > > >> > those
> > > > >> > > other systems simply implement Gremlin as language or do they
> use
> > > > >> > TinkerPop
> > > > >> > > for something / as a backend?
> > > > >> > >
> > > > >> >
> > > > >> > I'm sure Ahmad could answer but I'll quickly offer my take. I
> think
> > > > >> that in
> > > > >> > this case we should prefer "TinkerPop" over Gremlin as a
> top-level
> > > > name
> > > > >> > particularly because it's prefixed with "Apache" and there is no
> > > > "Apache
> > > > >> > Gremlin" which I tend to think is confusing when the words are
> that
> > > > >> close
> > > > >> > together. I can't recall over the years just how many times I've
> > > asked
> > > > >> for
> > > > >> > corrections in blog posts. :)
> > > > >> >
> > > > >> > Because that's a bit of a conceptual difference here. For
> example in
> > > > the
> > > > >> > > provider we are importing
> https://pypi.org/project/gremlinpython
> > > > not
> > > > >> > > "tinkerpop" - and it also does not have tinkerpop as
> dependency.
> > > > >> > >
> > > > >> >
> > > > >> > A bit of history goes along with a lot of our naming for what we
> > > term
> > > > >> > Gremlin Language Variants (GLVs), like gremlinpython, which are
> > > > >> variants of
> > > > >> > Gremlin natively implemented to allow users to express Gremlin
> in
> > > the
> > > > >> > idioms of their own language. They also provide driver
> connectivity
> > > to
> > > > >> > compatible servers. TinkerPop has mostly inherited all of its
> > > language
> > > > >> > variants, including gremlinpython which was the first, from
> > > > third-party
> > > > >> > community developers. As a project, we didn't really get a hand
> in
> > > the
> > > > >> > naming so with those projects already in heavy use we just
> kinda of
> > > > >> stuck
> > > > >> > to it and even doubled-down (like when we built gremlin-go
> within
> > > the
> > > > >> ASF).
> > > > >> >
> > > > >> > I think in this case, your project organization under "apache"
> seems
> > > > to
> > > > >> > almost lend itself nicely to apache/tinkerpop. i think users
> will
> > > > >> recognize
> > > > >> > it as equally as they recognize Gremlin.
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we
> > > should
> > > > >> ask
> > > > >> > > tinkerpop PMC what they think about it?
> > > > >> > >
> > > > >> >
> > > > >> > Gremlin is not an ASF trademark. That was debated for quite a
> long
> > > > time
> > > > >> > with trademarks@ along with deciding if Gremlin, the character
> and
> > > > his
> > > > >> > friends[1], were to be protected. In the end, for reasons I'm
> not
> > > > sure I
> > > > >> > quite remember, the ASF didn't think it was necessary.
> > > > >> >
> > > > >> > Anyway, I'm not sure if you noted my earlier post[2] but I'm
> one of
> > > > the
> > > > >> > original contributors to Apache TinkerPop, even before we
> brought it
> > > > to
> > > > >> the
> > > > >> > ASF so I'm pretty familiar with our project.  :)
> > > > >> >
> > > > >> > [1]
> > > > >> >
> > > > >> >
> > > > >>
> > > >
> > >
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> > > > >> > [2]
> > > https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> > > > >> >
> > > > >> >
> > > > >> > >
> > > > >> > >
> > > > >> > > J.
> > > > >> > >
> > > > >> > >
> > > > >> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> > > > >> spmallette@apache.org>
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > >
> > > > >> > > >
> > > > >> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > > >> > > > > Yeah . `apache/gremlin" seems like a better option then.
> Does
> > > > >> anyone
> > > > >> > > have
> > > > >> > > > > anything against it?
> > > > >> > > >
> > > > >> > > > In the interest of ASF trademarks, I would suggest it be
> called
> > > > >> > > > "apache/tinkerpop" with "Gremlin" naming reserved for
> operators
> > > > and
> > > > >> the
> > > > >> > > > like, as it is now with GremlinOperator. I think this makes
> > > sense
> > > > >> > because
> > > > >> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I
> > > would
> > > > >> > > > similarly suggest that references to "Apache Gremlin" and
> the
> > > like
> > > > >> > become
> > > > >> > > > "Apache TinkerPop".
> > > > >> > > >
> > > > >> > > > > I think we are pretty happy with accepting "other
> > > > >> > > > > apache" projects as providers, so I see no issue with
> Gremlin
> > > -
> > > > >> > knowing
> > > > >> > > > > that we can always reach out to our friendly Apache
> Community
> > > in
> > > > >> case
> > > > >> > > of
> > > > >> > > > > any issues. So - unless we do not hear any "opposition"
> in a
> > > few
> > > > >> > days,
> > > > >> > > I
> > > > >> > > > > think it would make sense if you start `[LAZY CONSENSUS]`
> > > > thread -
> > > > >> > > > > without a need for `[VOTE]` thread.
> > > > >> > > > >
> > > > >> > > > > One thing though that I would love to have - is to also
> have
> > > an
> > > > >> > > > integration
> > > > >> > > > > test if possible (we had it with apache.kafka for
> example) -
> > > > those
> > > > >> > are
> > > > >> > > > > tests that could run **some** graphdb database locally
> (via
> > > > >> > > > docker-compose)
> > > > >> > > > > and run a very rudimentary checks against a "real"
> database,
> > > > not a
> > > > >> > > mocked
> > > > >> > > > > call. That would make it more robust.
> > > > >> > > > >
> > > > >> > > > > More about integration tests, how to build, run, test
> them and
> > > > >> > > integrate
> > > > >> > > > > them in our CI can be found here:
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > >
> > >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > > >> > > > > - happy to help if you are stuck with it.
> > > > >> > > > >
> > > > >> > > > > J.
> > > > >> > > > >
> > > > >> > > > >
> > > > >> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > > > >> > > ahmad.farhan9715@gmail.com
> > > > >> > > > >
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > I pushed changes to move the provider into the “apache”
> > > > >> directory.
> > > > >> > > > After
> > > > >> > > > > > updating the class references across the project, I
> > > re-tested
> > > > >> and
> > > > >> > all
> > > > >> > > > tests
> > > > >> > > > > > passed.
> > > > >> > > > > >
> > > > >> > > > > > Regarding the use of Gremlin (or another graph query
> > > language
> > > > >> like
> > > > >> > > > Cypher
> > > > >> > > > > > and SPARQL) for a common package approach, here are my
> > > > thoughts
> > > > >> on
> > > > >> > > the
> > > > >> > > > pros
> > > > >> > > > > > and cons:
> > > > >> > > > > >
> > > > >> > > > > > pros (I can see only one):
> > > > >> > > > > >
> > > > >> > > > > >    - Gremlin has been widely adopted by different cloud
> > > > vendors
> > > > >> > (e.g.
> > > > >> > > > Azure
> > > > >> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as
> well as
> > > > in
> > > > >> > > > self-hosted
> > > > >> > > > > >    environments.
> > > > >> > > > > >
> > > > >> > > > > > cons:
> > > > >> > > > > >
> > > > >> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each
> have
> > > > >> their
> > > > >> > > own
> > > > >> > > > > >    drivers for executing queries.
> > > > >> > > > > >    - To achieve a common abstraction, a wrapper around
> each
> > > > >> driver
> > > > >> > > > would be
> > > > >> > > > > >    required. Each driver has its own connection
> parameters,
> > > > >> > > underlying
> > > > >> > > > > >    protocols, and may need method overrides for
> > > compatibility
> > > > >> with
> > > > >> > > > > > different
> > > > >> > > > > >    Python versions.
> > > > >> > > > > >    - Not all vendors support every query language; for
> > > > instance,
> > > > >> > > > Gremlin
> > > > >> > > > > >    for Neo4j has been deprecated in recent releases,
> while
> > > > >> Cosmos
> > > > >> > DB
> > > > >> > > > does
> > > > >> > > > > > not
> > > > >> > > > > >    support Cypher or SPARQL.
> > > > >> > > > > >
> > > > >> > > > > > While it would be ideal to have a unified graph query
> > > language
> > > > >> and
> > > > >> > > > driver
> > > > >> > > > > > that works seamlessly across different vendors, such a
> > > > solution
> > > > >> > does
> > > > >> > > > not
> > > > >> > > > > > exist at the moment. In my opinion, implementing
> > > > >> provider-specific
> > > > >> > > > > > solutions for each query language (Gremlin, Cypher,
> SPARQL)
> > > is
> > > > >> more
> > > > >> > > > > > realistic and practical given the current landscape.
> > > > >> > > > > >
> > > > >> > > > > > Happy to discuss further or answer any questions!
> > > > >> > > > > >
> > > > >> > > > > > Farhan
> > > > >> > > > > >
> > > > >> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > > > >> > > > ahmad.farhan9715@gmail.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > I have worked with two different graph database
> > > > vendors—Azure
> > > > >> > > Cosmos
> > > > >> > > > DB
> > > > >> > > > > > > and Neo4j. During our migration to Neo4j, we
> discovered
> > > that
> > > > >> > using
> > > > >> > > > the
> > > > >> > > > > > > Gremlin language wasn’t possible; we were forced to
> > > rewrite
> > > > >> all
> > > > >> > our
> > > > >> > > > > > queries
> > > > >> > > > > > > into Cypher, which is the native language for Neo4j
> and,
> > > in
> > > > my
> > > > >> > > > > > experience,
> > > > >> > > > > > > much simpler for querying.
> > > > >> > > > > > >
> > > > >> > > > > > > This situation highlights a key challenge for a common
> > > > >> > abstraction:
> > > > >> > > > the
> > > > >> > > > > > > underlying query languages and
> connection/authentication
> > > > >> > mechanisms
> > > > >> > > > vary
> > > > >> > > > > > > significantly. Gremlin is not only different from
> Cypher
> > > in
> > > > >> > syntax
> > > > >> > > > but is
> > > > >> > > > > > > also deprecated for Neo4j (see
> > > > >> > > > > > >
> > > > >> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> > > > >> > ).
> > > > >> > > > > > >
> > > > >> > > > > > > The question would be how can the common approach
> > > > accommodate
> > > > >> > these
> > > > >> > > > > > > different query languages?
> > > > >> > > > > > >
> > > > >> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
> > > > >> jarek@potiuk.com>
> > > > >> > > > wrote:
> > > > >> > > > > > >
> > > > >> > > > > > >> Without deep looking at the code I love the idea -
> it's
> > > > very
> > > > >> > > > similar to
> > > > >> > > > > > >> what we have for common.sql and common.io - and soon
> > > > >> > > > common.messaging
> > > > >> > > > > > - I
> > > > >> > > > > > >> also - long time ago - suggested common.dataframe
> that
> > > > >> someone
> > > > >> > > could
> > > > >> > > > > > >> submit
> > > > >> > > > > > >> using Apache Ibis:
> > > > >> > > > > > >>
> > > > >> >
> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > > > >> > > -
> > > > >> > > > > > >> similarly I believe there was an idea about
> common.llm
> > > ...
> > > > >> > > > > > >>
> > > > >> > > > > > >> I think the "common" pattern is a great one for
> Airflow,
> > > to
> > > > >> > build
> > > > >> > > > on top
> > > > >> > > > > > >> of
> > > > >> > > > > > >> "other giants" who build those common abstractions
> that
> > > you
> > > > >> can
> > > > >> > > > easily
> > > > >> > > > > > >> switch between different implementations of various
> data
> > > > >> access
> > > > >> > > > layers.
> > > > >> > > > > > >>
> > > > >> > > > > > >> My suggestion and question - would be however (not
> very
> > > > >> strong
> > > > >> > on
> > > > >> > > > it, I
> > > > >> > > > > > >> would love to hear what others think, I know it's
> been
> > > > >> somewhat
> > > > >> > > > > > >> contentious
> > > > >> > > > > > >> when I started the ibis discussion) - would be to
> make it
> > > > >> > > > > > "common.graph",
> > > > >> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> > > > >> > "apache.ibis"
> > > > >> > > -
> > > > >> > > > just
> > > > >> > > > > > >> to
> > > > >> > > > > > >> stress that those are not implementations of
> particular
> > > > >> service
> > > > >> > > but
> > > > >> > > > > > >> opinionated choice of particular technology to do
> > > "common"
> > > > >> > > > operations.
> > > > >> > > > > > >> This
> > > > >> > > > > > >> is what essentially "common.io" is . - it should be
> > > named
> > > > >> > > "fsspec"
> > > > >> > > > > > >> provider
> > > > >> > > > > > >> if we were to name it by the "library" that
> implemented
> > > it.
> > > > >> > > > > > >>
> > > > >> > > > > > >> J.
> > > > >> > > > > > >>
> > > > >> > > > > > >>
> > > > >> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > > >> > > > > > ahmad.farhan9715@gmail.com>
> > > > >> > > > > > >> wrote:
> > > > >> > > > > > >>
> > > > >> > > > > > >> > Hi Everyone,
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > I’ve created a draft PR (
> > > > >> > > > https://github.com/apache/airflow/pull/46977
> > > > >> > > > > > )
> > > > >> > > > > > >> to
> > > > >> > > > > > >> > introduce and discuss a new provider for using
> > > > Gremlin—the
> > > > >> > graph
> > > > >> > > > > > >> traversal
> > > > >> > > > > > >> > language of Apache TinkerPop (more details here:
> > > > >> > > > > > >> > https://tinkerpop.apache.org/gremlin.html).
> Gremlin is
> > > > >> > > supported
> > > > >> > > > by
> > > > >> > > > > > >> > various
> > > > >> > > > > > >> > graph database vendors such as Azure Cosmos DB and
> > > Amazon
> > > > >> > > Neptune.
> > > > >> > > > > > >> > Previously, I had to develop a custom hook to query
> > > data
> > > > >> from
> > > > >> > > > Azure
> > > > >> > > > > > >> Cosmos
> > > > >> > > > > > >> > DB using Apache Gremlin.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > I managed to create a provider and run it locally
> on
> > > the
> > > > >> main
> > > > >> > > > branch.
> > > > >> > > > > > >> > However, I ran into the BaseHook issue (
> > > > >> > > > > > >> > https://github.com/apache/airflow/issues/45233) on
> > > that
> > > > >> > branch,
> > > > >> > > > so I
> > > > >> > > > > > >> ended
> > > > >> > > > > > >> > up testing it fully on the v2-10-test branch. The
> PR
> > > > >> should be
> > > > >> > > > > > complete,
> > > > >> > > > > > >> > but I’ve kept it as a draft for now while we
> discuss
> > > the
> > > > >> > > provider.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > I’m a new contributor, so I’m especially eager to
> hear
> > > > your
> > > > >> > > > feedback.
> > > > >> > > > > > >> > Comments on the PR is very welcome, and please feel
> > > free
> > > > to
> > > > >> > > reach
> > > > >> > > > out
> > > > >> > > > > > >> with
> > > > >> > > > > > >> > any questions via email or Slack.
> > > > >> > > > > > >> >
> > > > >> > > > > > >> > Thanks,
> > > > >> > > > > > >> > Ahmad Farhan
> > > > >> > > > > > >> >
> > > > >> > > > > > >>
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > > >
> > > > >>
> ---------------------------------------------------------------------
> > > > >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > >> > > > For additional commands, e-mail:
> dev-help@airflow.apache.org
> > > > >> > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] New Provider for Gremlin

[Paul King <pa...@apache.org>]

Nice to see the Apache TinkerPop/Gremlin support. As well as using Gremlin with TinkerPop's TinkerGraph, I have also used it with OrientDB, ArcadeDB, and Apache HugeGraph. If anyone is interested, I did a blog post here: https://groovy.apache.org/blog/groovy-graph-databases
(Sorry but it's Groovy not Python)

Cheers, Paul.

On 2025/03/08 11:34:11 Ahmad Farhan wrote:
> Hi!
> I managed to do the last cleanup on the PR, and is ready for thorough
> review. I will create a  `[LAZY CONSENSUS]` thread on Monday/Tuesday. Feel
> free to review and comment on the PR in the meantime.
> 
> Farhan
> 
> On Thu, Mar 6, 2025 at 1:40 PM Stephen Mallette <sp...@gmail.com>
> wrote:
> 
> > Hi Ahmad, thanks for the updates! I've spent some time looking at the
> > changes and have added some comments and questions. I've pointed out a
> > couple things for future work that I think will be important, but nothing
> > that needs to be changed for this PR in my mind. Looking forward to seeing
> > your responses, other community feedback, and ultimately a merge of the
> > provider - take care!
> >
> > On Thu, Mar 6, 2025 at 7:28 AM Ahmad Farhan <ah...@gmail.com>
> > wrote:
> >
> > > Hello again!
> > > After several days of debugging and branch restructuring I managed to
> > > create another PR https://github.com/apache/airflow/pull/47446, this is
> > > due
> > > to major merge conflicts after the www clean up.
> > > The renaming of the provider to apache-tinkerpop is done but I kept the
> > > integration testing as 'gremlin' to avoid the conflict/confusion with the
> > > server name within the CI.
> > >
> > > Please review the PR thoroughly and I am happy to work on the following
> > > steps.
> > >
> > > Thanks,
> > > Farhan
> > >
> > > On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <ahmad.farhan9715@gmail.com
> > >
> > > wrote:
> > >
> > > > I did look into the naming and I thought that it would need to be
> > > > discussed at some point before the lazy consensus stage after the dev
> > > work
> > > > is done but I guess I was wrong :)
> > > >
> > > > I read through some docs regarding the naming and I kept thinking that
> > > > Apache Gremlin might not be right, so I decided to remove 'Apache' from
> > > all
> > > > doc strings. One thing that popped in one of the documentations from
> > > > Microsoft that says Apache Gremlin (here
> > > > https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction
> > )
> > > > which confused me.
> > > >
> > > > I will change the folder to apache/tinkerpop. Thanks Stephen for the
> > > > in-depth explanation.
> > > >
> > > > On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > > >
> > > >> Cool. I will let Ahmad comment, but I think we found the **someone**
> > who
> > > >> will help in case there are some future issues with the
> > > Tinkerpop/Gremlin
> > > >> provider.
> > > >> While I like Gremlin better (it's just a cool name and I like the
> > logo,
> > > >> tinkerpop has a cool logo as well
> > > https://tinkerpop.apache.org/index.html
> > > >> ).
> > > >>
> > > >> So as long as we decide not to use common.graph -> I am fine with both
> > > :)
> > > >>
> > > >> j.
> > > >>
> > > >>
> > > >> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <
> > spmallette@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > >> >
> > > >> > > > In the interest of ASF trademarks, I would suggest it be called
> > > >> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators
> > and
> > > >> the
> > > >> > > like, as it is now with GremlinOperator. I think this makes sense
> > > >> because
> > > >> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > > >> > > similarly suggest that references to "Apache Gremlin" and the like
> > > >> become
> > > >> > > "Apache TinkerPop".
> > > >> > >
> > > >> > > That's an interesting one - indeed TinkerPop is the PMC/
> > Framework -
> > > >> > > Gremlin is the language.
> > > >> > >
> > > >> > > I am not sure we are actually using TinkerPop here - because
> > > >> TinkerPop is
> > > >> > > the whole framework - Ahmad, can you explain the relation there -
> > > are
> > > >> > those
> > > >> > > other systems simply implement Gremlin as language or do they use
> > > >> > TinkerPop
> > > >> > > for something / as a backend?
> > > >> > >
> > > >> >
> > > >> > I'm sure Ahmad could answer but I'll quickly offer my take. I think
> > > >> that in
> > > >> > this case we should prefer "TinkerPop" over Gremlin as a top-level
> > > name
> > > >> > particularly because it's prefixed with "Apache" and there is no
> > > "Apache
> > > >> > Gremlin" which I tend to think is confusing when the words are that
> > > >> close
> > > >> > together. I can't recall over the years just how many times I've
> > asked
> > > >> for
> > > >> > corrections in blog posts. :)
> > > >> >
> > > >> > Because that's a bit of a conceptual difference here. For example in
> > > the
> > > >> > > provider we are importing https://pypi.org/project/gremlinpython
> > > not
> > > >> > > "tinkerpop" - and it also does not have tinkerpop as dependency.
> > > >> > >
> > > >> >
> > > >> > A bit of history goes along with a lot of our naming for what we
> > term
> > > >> > Gremlin Language Variants (GLVs), like gremlinpython, which are
> > > >> variants of
> > > >> > Gremlin natively implemented to allow users to express Gremlin in
> > the
> > > >> > idioms of their own language. They also provide driver connectivity
> > to
> > > >> > compatible servers. TinkerPop has mostly inherited all of its
> > language
> > > >> > variants, including gremlinpython which was the first, from
> > > third-party
> > > >> > community developers. As a project, we didn't really get a hand in
> > the
> > > >> > naming so with those projects already in heavy use we just kinda of
> > > >> stuck
> > > >> > to it and even doubled-down (like when we built gremlin-go within
> > the
> > > >> ASF).
> > > >> >
> > > >> > I think in this case, your project organization under "apache" seems
> > > to
> > > >> > almost lend itself nicely to apache/tinkerpop. i think users will
> > > >> recognize
> > > >> > it as equally as they recognize Gremlin.
> > > >> >
> > > >> >
> > > >> > >
> > > >> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we
> > should
> > > >> ask
> > > >> > > tinkerpop PMC what they think about it?
> > > >> > >
> > > >> >
> > > >> > Gremlin is not an ASF trademark. That was debated for quite a long
> > > time
> > > >> > with trademarks@ along with deciding if Gremlin, the character and
> > > his
> > > >> > friends[1], were to be protected. In the end, for reasons I'm not
> > > sure I
> > > >> > quite remember, the ASF didn't think it was necessary.
> > > >> >
> > > >> > Anyway, I'm not sure if you noted my earlier post[2] but I'm one of
> > > the
> > > >> > original contributors to Apache TinkerPop, even before we brought it
> > > to
> > > >> the
> > > >> > ASF so I'm pretty familiar with our project.  :)
> > > >> >
> > > >> > [1]
> > > >> >
> > > >> >
> > > >>
> > >
> > https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> > > >> > [2]
> > https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> > > >> >
> > > >> >
> > > >> > >
> > > >> > >
> > > >> > > J.
> > > >> > >
> > > >> > >
> > > >> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> > > >> spmallette@apache.org>
> > > >> > > wrote:
> > > >> > >
> > > >> > > >
> > > >> > > >
> > > >> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > >> > > > > Yeah . `apache/gremlin" seems like a better option then. Does
> > > >> anyone
> > > >> > > have
> > > >> > > > > anything against it?
> > > >> > > >
> > > >> > > > In the interest of ASF trademarks, I would suggest it be called
> > > >> > > > "apache/tinkerpop" with "Gremlin" naming reserved for operators
> > > and
> > > >> the
> > > >> > > > like, as it is now with GremlinOperator. I think this makes
> > sense
> > > >> > because
> > > >> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I
> > would
> > > >> > > > similarly suggest that references to "Apache Gremlin" and the
> > like
> > > >> > become
> > > >> > > > "Apache TinkerPop".
> > > >> > > >
> > > >> > > > > I think we are pretty happy with accepting "other
> > > >> > > > > apache" projects as providers, so I see no issue with Gremlin
> > -
> > > >> > knowing
> > > >> > > > > that we can always reach out to our friendly Apache Community
> > in
> > > >> case
> > > >> > > of
> > > >> > > > > any issues. So - unless we do not hear any "opposition" in a
> > few
> > > >> > days,
> > > >> > > I
> > > >> > > > > think it would make sense if you start `[LAZY CONSENSUS]`
> > > thread -
> > > >> > > > > without a need for `[VOTE]` thread.
> > > >> > > > >
> > > >> > > > > One thing though that I would love to have - is to also have
> > an
> > > >> > > > integration
> > > >> > > > > test if possible (we had it with apache.kafka for example) -
> > > those
> > > >> > are
> > > >> > > > > tests that could run **some** graphdb database locally (via
> > > >> > > > docker-compose)
> > > >> > > > > and run a very rudimentary checks against a "real" database,
> > > not a
> > > >> > > mocked
> > > >> > > > > call. That would make it more robust.
> > > >> > > > >
> > > >> > > > > More about integration tests, how to build, run, test them and
> > > >> > > integrate
> > > >> > > > > them in our CI can be found here:
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> > https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > >> > > > > - happy to help if you are stuck with it.
> > > >> > > > >
> > > >> > > > > J.
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > > >> > > ahmad.farhan9715@gmail.com
> > > >> > > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > I pushed changes to move the provider into the “apache”
> > > >> directory.
> > > >> > > > After
> > > >> > > > > > updating the class references across the project, I
> > re-tested
> > > >> and
> > > >> > all
> > > >> > > > tests
> > > >> > > > > > passed.
> > > >> > > > > >
> > > >> > > > > > Regarding the use of Gremlin (or another graph query
> > language
> > > >> like
> > > >> > > > Cypher
> > > >> > > > > > and SPARQL) for a common package approach, here are my
> > > thoughts
> > > >> on
> > > >> > > the
> > > >> > > > pros
> > > >> > > > > > and cons:
> > > >> > > > > >
> > > >> > > > > > pros (I can see only one):
> > > >> > > > > >
> > > >> > > > > >    - Gremlin has been widely adopted by different cloud
> > > vendors
> > > >> > (e.g.
> > > >> > > > Azure
> > > >> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as
> > > in
> > > >> > > > self-hosted
> > > >> > > > > >    environments.
> > > >> > > > > >
> > > >> > > > > > cons:
> > > >> > > > > >
> > > >> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have
> > > >> their
> > > >> > > own
> > > >> > > > > >    drivers for executing queries.
> > > >> > > > > >    - To achieve a common abstraction, a wrapper around each
> > > >> driver
> > > >> > > > would be
> > > >> > > > > >    required. Each driver has its own connection parameters,
> > > >> > > underlying
> > > >> > > > > >    protocols, and may need method overrides for
> > compatibility
> > > >> with
> > > >> > > > > > different
> > > >> > > > > >    Python versions.
> > > >> > > > > >    - Not all vendors support every query language; for
> > > instance,
> > > >> > > > Gremlin
> > > >> > > > > >    for Neo4j has been deprecated in recent releases, while
> > > >> Cosmos
> > > >> > DB
> > > >> > > > does
> > > >> > > > > > not
> > > >> > > > > >    support Cypher or SPARQL.
> > > >> > > > > >
> > > >> > > > > > While it would be ideal to have a unified graph query
> > language
> > > >> and
> > > >> > > > driver
> > > >> > > > > > that works seamlessly across different vendors, such a
> > > solution
> > > >> > does
> > > >> > > > not
> > > >> > > > > > exist at the moment. In my opinion, implementing
> > > >> provider-specific
> > > >> > > > > > solutions for each query language (Gremlin, Cypher, SPARQL)
> > is
> > > >> more
> > > >> > > > > > realistic and practical given the current landscape.
> > > >> > > > > >
> > > >> > > > > > Happy to discuss further or answer any questions!
> > > >> > > > > >
> > > >> > > > > > Farhan
> > > >> > > > > >
> > > >> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > > >> > > > ahmad.farhan9715@gmail.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > I have worked with two different graph database
> > > vendors—Azure
> > > >> > > Cosmos
> > > >> > > > DB
> > > >> > > > > > > and Neo4j. During our migration to Neo4j, we discovered
> > that
> > > >> > using
> > > >> > > > the
> > > >> > > > > > > Gremlin language wasn’t possible; we were forced to
> > rewrite
> > > >> all
> > > >> > our
> > > >> > > > > > queries
> > > >> > > > > > > into Cypher, which is the native language for Neo4j and,
> > in
> > > my
> > > >> > > > > > experience,
> > > >> > > > > > > much simpler for querying.
> > > >> > > > > > >
> > > >> > > > > > > This situation highlights a key challenge for a common
> > > >> > abstraction:
> > > >> > > > the
> > > >> > > > > > > underlying query languages and connection/authentication
> > > >> > mechanisms
> > > >> > > > vary
> > > >> > > > > > > significantly. Gremlin is not only different from Cypher
> > in
> > > >> > syntax
> > > >> > > > but is
> > > >> > > > > > > also deprecated for Neo4j (see
> > > >> > > > > > >
> > > >> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> > > >> > ).
> > > >> > > > > > >
> > > >> > > > > > > The question would be how can the common approach
> > > accommodate
> > > >> > these
> > > >> > > > > > > different query languages?
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
> > > >> jarek@potiuk.com>
> > > >> > > > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Without deep looking at the code I love the idea - it's
> > > very
> > > >> > > > similar to
> > > >> > > > > > >> what we have for common.sql and common.io - and soon
> > > >> > > > common.messaging
> > > >> > > > > > - I
> > > >> > > > > > >> also - long time ago - suggested common.dataframe that
> > > >> someone
> > > >> > > could
> > > >> > > > > > >> submit
> > > >> > > > > > >> using Apache Ibis:
> > > >> > > > > > >>
> > > >> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > > >> > > -
> > > >> > > > > > >> similarly I believe there was an idea about common.llm
> > ...
> > > >> > > > > > >>
> > > >> > > > > > >> I think the "common" pattern is a great one for Airflow,
> > to
> > > >> > build
> > > >> > > > on top
> > > >> > > > > > >> of
> > > >> > > > > > >> "other giants" who build those common abstractions that
> > you
> > > >> can
> > > >> > > > easily
> > > >> > > > > > >> switch between different implementations of various data
> > > >> access
> > > >> > > > layers.
> > > >> > > > > > >>
> > > >> > > > > > >> My suggestion and question - would be however (not very
> > > >> strong
> > > >> > on
> > > >> > > > it, I
> > > >> > > > > > >> would love to hear what others think, I know it's been
> > > >> somewhat
> > > >> > > > > > >> contentious
> > > >> > > > > > >> when I started the ibis discussion) - would be to make it
> > > >> > > > > > "common.graph",
> > > >> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> > > >> > "apache.ibis"
> > > >> > > -
> > > >> > > > just
> > > >> > > > > > >> to
> > > >> > > > > > >> stress that those are not implementations of particular
> > > >> service
> > > >> > > but
> > > >> > > > > > >> opinionated choice of particular technology to do
> > "common"
> > > >> > > > operations.
> > > >> > > > > > >> This
> > > >> > > > > > >> is what essentially "common.io" is . - it should be
> > named
> > > >> > > "fsspec"
> > > >> > > > > > >> provider
> > > >> > > > > > >> if we were to name it by the "library" that implemented
> > it.
> > > >> > > > > > >>
> > > >> > > > > > >> J.
> > > >> > > > > > >>
> > > >> > > > > > >>
> > > >> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > >> > > > > > ahmad.farhan9715@gmail.com>
> > > >> > > > > > >> wrote:
> > > >> > > > > > >>
> > > >> > > > > > >> > Hi Everyone,
> > > >> > > > > > >> >
> > > >> > > > > > >> > I’ve created a draft PR (
> > > >> > > > https://github.com/apache/airflow/pull/46977
> > > >> > > > > > )
> > > >> > > > > > >> to
> > > >> > > > > > >> > introduce and discuss a new provider for using
> > > Gremlin—the
> > > >> > graph
> > > >> > > > > > >> traversal
> > > >> > > > > > >> > language of Apache TinkerPop (more details here:
> > > >> > > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> > > >> > > supported
> > > >> > > > by
> > > >> > > > > > >> > various
> > > >> > > > > > >> > graph database vendors such as Azure Cosmos DB and
> > Amazon
> > > >> > > Neptune.
> > > >> > > > > > >> > Previously, I had to develop a custom hook to query
> > data
> > > >> from
> > > >> > > > Azure
> > > >> > > > > > >> Cosmos
> > > >> > > > > > >> > DB using Apache Gremlin.
> > > >> > > > > > >> >
> > > >> > > > > > >> > I managed to create a provider and run it locally on
> > the
> > > >> main
> > > >> > > > branch.
> > > >> > > > > > >> > However, I ran into the BaseHook issue (
> > > >> > > > > > >> > https://github.com/apache/airflow/issues/45233) on
> > that
> > > >> > branch,
> > > >> > > > so I
> > > >> > > > > > >> ended
> > > >> > > > > > >> > up testing it fully on the v2-10-test branch. The PR
> > > >> should be
> > > >> > > > > > complete,
> > > >> > > > > > >> > but I’ve kept it as a draft for now while we discuss
> > the
> > > >> > > provider.
> > > >> > > > > > >> >
> > > >> > > > > > >> > I’m a new contributor, so I’m especially eager to hear
> > > your
> > > >> > > > feedback.
> > > >> > > > > > >> > Comments on the PR is very welcome, and please feel
> > free
> > > to
> > > >> > > reach
> > > >> > > > out
> > > >> > > > > > >> with
> > > >> > > > > > >> > any questions via email or Slack.
> > > >> > > > > > >> >
> > > >> > > > > > >> > Thanks,
> > > >> > > > > > >> > Ahmad Farhan
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > > >
> > > >> ---------------------------------------------------------------------
> > > >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > >> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

Hi!
I managed to do the last cleanup on the PR, and is ready for thorough
review. I will create a  `[LAZY CONSENSUS]` thread on Monday/Tuesday. Feel
free to review and comment on the PR in the meantime.

Farhan

On Thu, Mar 6, 2025 at 1:40 PM Stephen Mallette <sp...@gmail.com>
wrote:

> Hi Ahmad, thanks for the updates! I've spent some time looking at the
> changes and have added some comments and questions. I've pointed out a
> couple things for future work that I think will be important, but nothing
> that needs to be changed for this PR in my mind. Looking forward to seeing
> your responses, other community feedback, and ultimately a merge of the
> provider - take care!
>
> On Thu, Mar 6, 2025 at 7:28 AM Ahmad Farhan <ah...@gmail.com>
> wrote:
>
> > Hello again!
> > After several days of debugging and branch restructuring I managed to
> > create another PR https://github.com/apache/airflow/pull/47446, this is
> > due
> > to major merge conflicts after the www clean up.
> > The renaming of the provider to apache-tinkerpop is done but I kept the
> > integration testing as 'gremlin' to avoid the conflict/confusion with the
> > server name within the CI.
> >
> > Please review the PR thoroughly and I am happy to work on the following
> > steps.
> >
> > Thanks,
> > Farhan
> >
> > On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <ahmad.farhan9715@gmail.com
> >
> > wrote:
> >
> > > I did look into the naming and I thought that it would need to be
> > > discussed at some point before the lazy consensus stage after the dev
> > work
> > > is done but I guess I was wrong :)
> > >
> > > I read through some docs regarding the naming and I kept thinking that
> > > Apache Gremlin might not be right, so I decided to remove 'Apache' from
> > all
> > > doc strings. One thing that popped in one of the documentations from
> > > Microsoft that says Apache Gremlin (here
> > > https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction
> )
> > > which confused me.
> > >
> > > I will change the folder to apache/tinkerpop. Thanks Stephen for the
> > > in-depth explanation.
> > >
> > > On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> Cool. I will let Ahmad comment, but I think we found the **someone**
> who
> > >> will help in case there are some future issues with the
> > Tinkerpop/Gremlin
> > >> provider.
> > >> While I like Gremlin better (it's just a cool name and I like the
> logo,
> > >> tinkerpop has a cool logo as well
> > https://tinkerpop.apache.org/index.html
> > >> ).
> > >>
> > >> So as long as we decide not to use common.graph -> I am fine with both
> > :)
> > >>
> > >> j.
> > >>
> > >>
> > >> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <
> spmallette@gmail.com>
> > >> wrote:
> > >>
> > >> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > >> >
> > >> > > > In the interest of ASF trademarks, I would suggest it be called
> > >> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators
> and
> > >> the
> > >> > > like, as it is now with GremlinOperator. I think this makes sense
> > >> because
> > >> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > >> > > similarly suggest that references to "Apache Gremlin" and the like
> > >> become
> > >> > > "Apache TinkerPop".
> > >> > >
> > >> > > That's an interesting one - indeed TinkerPop is the PMC/
> Framework -
> > >> > > Gremlin is the language.
> > >> > >
> > >> > > I am not sure we are actually using TinkerPop here - because
> > >> TinkerPop is
> > >> > > the whole framework - Ahmad, can you explain the relation there -
> > are
> > >> > those
> > >> > > other systems simply implement Gremlin as language or do they use
> > >> > TinkerPop
> > >> > > for something / as a backend?
> > >> > >
> > >> >
> > >> > I'm sure Ahmad could answer but I'll quickly offer my take. I think
> > >> that in
> > >> > this case we should prefer "TinkerPop" over Gremlin as a top-level
> > name
> > >> > particularly because it's prefixed with "Apache" and there is no
> > "Apache
> > >> > Gremlin" which I tend to think is confusing when the words are that
> > >> close
> > >> > together. I can't recall over the years just how many times I've
> asked
> > >> for
> > >> > corrections in blog posts. :)
> > >> >
> > >> > Because that's a bit of a conceptual difference here. For example in
> > the
> > >> > > provider we are importing https://pypi.org/project/gremlinpython
> > not
> > >> > > "tinkerpop" - and it also does not have tinkerpop as dependency.
> > >> > >
> > >> >
> > >> > A bit of history goes along with a lot of our naming for what we
> term
> > >> > Gremlin Language Variants (GLVs), like gremlinpython, which are
> > >> variants of
> > >> > Gremlin natively implemented to allow users to express Gremlin in
> the
> > >> > idioms of their own language. They also provide driver connectivity
> to
> > >> > compatible servers. TinkerPop has mostly inherited all of its
> language
> > >> > variants, including gremlinpython which was the first, from
> > third-party
> > >> > community developers. As a project, we didn't really get a hand in
> the
> > >> > naming so with those projects already in heavy use we just kinda of
> > >> stuck
> > >> > to it and even doubled-down (like when we built gremlin-go within
> the
> > >> ASF).
> > >> >
> > >> > I think in this case, your project organization under "apache" seems
> > to
> > >> > almost lend itself nicely to apache/tinkerpop. i think users will
> > >> recognize
> > >> > it as equally as they recognize Gremlin.
> > >> >
> > >> >
> > >> > >
> > >> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we
> should
> > >> ask
> > >> > > tinkerpop PMC what they think about it?
> > >> > >
> > >> >
> > >> > Gremlin is not an ASF trademark. That was debated for quite a long
> > time
> > >> > with trademarks@ along with deciding if Gremlin, the character and
> > his
> > >> > friends[1], were to be protected. In the end, for reasons I'm not
> > sure I
> > >> > quite remember, the ASF didn't think it was necessary.
> > >> >
> > >> > Anyway, I'm not sure if you noted my earlier post[2] but I'm one of
> > the
> > >> > original contributors to Apache TinkerPop, even before we brought it
> > to
> > >> the
> > >> > ASF so I'm pretty familiar with our project.  :)
> > >> >
> > >> > [1]
> > >> >
> > >> >
> > >>
> >
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> > >> > [2]
> https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> > >> >
> > >> >
> > >> > >
> > >> > >
> > >> > > J.
> > >> > >
> > >> > >
> > >> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> > >> spmallette@apache.org>
> > >> > > wrote:
> > >> > >
> > >> > > >
> > >> > > >
> > >> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > >> > > > > Yeah . `apache/gremlin" seems like a better option then. Does
> > >> anyone
> > >> > > have
> > >> > > > > anything against it?
> > >> > > >
> > >> > > > In the interest of ASF trademarks, I would suggest it be called
> > >> > > > "apache/tinkerpop" with "Gremlin" naming reserved for operators
> > and
> > >> the
> > >> > > > like, as it is now with GremlinOperator. I think this makes
> sense
> > >> > because
> > >> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I
> would
> > >> > > > similarly suggest that references to "Apache Gremlin" and the
> like
> > >> > become
> > >> > > > "Apache TinkerPop".
> > >> > > >
> > >> > > > > I think we are pretty happy with accepting "other
> > >> > > > > apache" projects as providers, so I see no issue with Gremlin
> -
> > >> > knowing
> > >> > > > > that we can always reach out to our friendly Apache Community
> in
> > >> case
> > >> > > of
> > >> > > > > any issues. So - unless we do not hear any "opposition" in a
> few
> > >> > days,
> > >> > > I
> > >> > > > > think it would make sense if you start `[LAZY CONSENSUS]`
> > thread -
> > >> > > > > without a need for `[VOTE]` thread.
> > >> > > > >
> > >> > > > > One thing though that I would love to have - is to also have
> an
> > >> > > > integration
> > >> > > > > test if possible (we had it with apache.kafka for example) -
> > those
> > >> > are
> > >> > > > > tests that could run **some** graphdb database locally (via
> > >> > > > docker-compose)
> > >> > > > > and run a very rudimentary checks against a "real" database,
> > not a
> > >> > > mocked
> > >> > > > > call. That would make it more robust.
> > >> > > > >
> > >> > > > > More about integration tests, how to build, run, test them and
> > >> > > integrate
> > >> > > > > them in our CI can be found here:
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > >> > > > > - happy to help if you are stuck with it.
> > >> > > > >
> > >> > > > > J.
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > >> > > ahmad.farhan9715@gmail.com
> > >> > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > I pushed changes to move the provider into the “apache”
> > >> directory.
> > >> > > > After
> > >> > > > > > updating the class references across the project, I
> re-tested
> > >> and
> > >> > all
> > >> > > > tests
> > >> > > > > > passed.
> > >> > > > > >
> > >> > > > > > Regarding the use of Gremlin (or another graph query
> language
> > >> like
> > >> > > > Cypher
> > >> > > > > > and SPARQL) for a common package approach, here are my
> > thoughts
> > >> on
> > >> > > the
> > >> > > > pros
> > >> > > > > > and cons:
> > >> > > > > >
> > >> > > > > > pros (I can see only one):
> > >> > > > > >
> > >> > > > > >    - Gremlin has been widely adopted by different cloud
> > vendors
> > >> > (e.g.
> > >> > > > Azure
> > >> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as
> > in
> > >> > > > self-hosted
> > >> > > > > >    environments.
> > >> > > > > >
> > >> > > > > > cons:
> > >> > > > > >
> > >> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have
> > >> their
> > >> > > own
> > >> > > > > >    drivers for executing queries.
> > >> > > > > >    - To achieve a common abstraction, a wrapper around each
> > >> driver
> > >> > > > would be
> > >> > > > > >    required. Each driver has its own connection parameters,
> > >> > > underlying
> > >> > > > > >    protocols, and may need method overrides for
> compatibility
> > >> with
> > >> > > > > > different
> > >> > > > > >    Python versions.
> > >> > > > > >    - Not all vendors support every query language; for
> > instance,
> > >> > > > Gremlin
> > >> > > > > >    for Neo4j has been deprecated in recent releases, while
> > >> Cosmos
> > >> > DB
> > >> > > > does
> > >> > > > > > not
> > >> > > > > >    support Cypher or SPARQL.
> > >> > > > > >
> > >> > > > > > While it would be ideal to have a unified graph query
> language
> > >> and
> > >> > > > driver
> > >> > > > > > that works seamlessly across different vendors, such a
> > solution
> > >> > does
> > >> > > > not
> > >> > > > > > exist at the moment. In my opinion, implementing
> > >> provider-specific
> > >> > > > > > solutions for each query language (Gremlin, Cypher, SPARQL)
> is
> > >> more
> > >> > > > > > realistic and practical given the current landscape.
> > >> > > > > >
> > >> > > > > > Happy to discuss further or answer any questions!
> > >> > > > > >
> > >> > > > > > Farhan
> > >> > > > > >
> > >> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > >> > > > ahmad.farhan9715@gmail.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > I have worked with two different graph database
> > vendors—Azure
> > >> > > Cosmos
> > >> > > > DB
> > >> > > > > > > and Neo4j. During our migration to Neo4j, we discovered
> that
> > >> > using
> > >> > > > the
> > >> > > > > > > Gremlin language wasn’t possible; we were forced to
> rewrite
> > >> all
> > >> > our
> > >> > > > > > queries
> > >> > > > > > > into Cypher, which is the native language for Neo4j and,
> in
> > my
> > >> > > > > > experience,
> > >> > > > > > > much simpler for querying.
> > >> > > > > > >
> > >> > > > > > > This situation highlights a key challenge for a common
> > >> > abstraction:
> > >> > > > the
> > >> > > > > > > underlying query languages and connection/authentication
> > >> > mechanisms
> > >> > > > vary
> > >> > > > > > > significantly. Gremlin is not only different from Cypher
> in
> > >> > syntax
> > >> > > > but is
> > >> > > > > > > also deprecated for Neo4j (see
> > >> > > > > > >
> > >> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> > >> > ).
> > >> > > > > > >
> > >> > > > > > > The question would be how can the common approach
> > accommodate
> > >> > these
> > >> > > > > > > different query languages?
> > >> > > > > > >
> > >> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
> > >> jarek@potiuk.com>
> > >> > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> Without deep looking at the code I love the idea - it's
> > very
> > >> > > > similar to
> > >> > > > > > >> what we have for common.sql and common.io - and soon
> > >> > > > common.messaging
> > >> > > > > > - I
> > >> > > > > > >> also - long time ago - suggested common.dataframe that
> > >> someone
> > >> > > could
> > >> > > > > > >> submit
> > >> > > > > > >> using Apache Ibis:
> > >> > > > > > >>
> > >> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > >> > > -
> > >> > > > > > >> similarly I believe there was an idea about common.llm
> ...
> > >> > > > > > >>
> > >> > > > > > >> I think the "common" pattern is a great one for Airflow,
> to
> > >> > build
> > >> > > > on top
> > >> > > > > > >> of
> > >> > > > > > >> "other giants" who build those common abstractions that
> you
> > >> can
> > >> > > > easily
> > >> > > > > > >> switch between different implementations of various data
> > >> access
> > >> > > > layers.
> > >> > > > > > >>
> > >> > > > > > >> My suggestion and question - would be however (not very
> > >> strong
> > >> > on
> > >> > > > it, I
> > >> > > > > > >> would love to hear what others think, I know it's been
> > >> somewhat
> > >> > > > > > >> contentious
> > >> > > > > > >> when I started the ibis discussion) - would be to make it
> > >> > > > > > "common.graph",
> > >> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> > >> > "apache.ibis"
> > >> > > -
> > >> > > > just
> > >> > > > > > >> to
> > >> > > > > > >> stress that those are not implementations of particular
> > >> service
> > >> > > but
> > >> > > > > > >> opinionated choice of particular technology to do
> "common"
> > >> > > > operations.
> > >> > > > > > >> This
> > >> > > > > > >> is what essentially "common.io" is . - it should be
> named
> > >> > > "fsspec"
> > >> > > > > > >> provider
> > >> > > > > > >> if we were to name it by the "library" that implemented
> it.
> > >> > > > > > >>
> > >> > > > > > >> J.
> > >> > > > > > >>
> > >> > > > > > >>
> > >> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > >> > > > > > ahmad.farhan9715@gmail.com>
> > >> > > > > > >> wrote:
> > >> > > > > > >>
> > >> > > > > > >> > Hi Everyone,
> > >> > > > > > >> >
> > >> > > > > > >> > I’ve created a draft PR (
> > >> > > > https://github.com/apache/airflow/pull/46977
> > >> > > > > > )
> > >> > > > > > >> to
> > >> > > > > > >> > introduce and discuss a new provider for using
> > Gremlin—the
> > >> > graph
> > >> > > > > > >> traversal
> > >> > > > > > >> > language of Apache TinkerPop (more details here:
> > >> > > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> > >> > > supported
> > >> > > > by
> > >> > > > > > >> > various
> > >> > > > > > >> > graph database vendors such as Azure Cosmos DB and
> Amazon
> > >> > > Neptune.
> > >> > > > > > >> > Previously, I had to develop a custom hook to query
> data
> > >> from
> > >> > > > Azure
> > >> > > > > > >> Cosmos
> > >> > > > > > >> > DB using Apache Gremlin.
> > >> > > > > > >> >
> > >> > > > > > >> > I managed to create a provider and run it locally on
> the
> > >> main
> > >> > > > branch.
> > >> > > > > > >> > However, I ran into the BaseHook issue (
> > >> > > > > > >> > https://github.com/apache/airflow/issues/45233) on
> that
> > >> > branch,
> > >> > > > so I
> > >> > > > > > >> ended
> > >> > > > > > >> > up testing it fully on the v2-10-test branch. The PR
> > >> should be
> > >> > > > > > complete,
> > >> > > > > > >> > but I’ve kept it as a draft for now while we discuss
> the
> > >> > > provider.
> > >> > > > > > >> >
> > >> > > > > > >> > I’m a new contributor, so I’m especially eager to hear
> > your
> > >> > > > feedback.
> > >> > > > > > >> > Comments on the PR is very welcome, and please feel
> free
> > to
> > >> > > reach
> > >> > > > out
> > >> > > > > > >> with
> > >> > > > > > >> > any questions via email or Slack.
> > >> > > > > > >> >
> > >> > > > > > >> > Thanks,
> > >> > > > > > >> > Ahmad Farhan
> > >> > > > > > >> >
> > >> > > > > > >>
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > > >
> > >> ---------------------------------------------------------------------
> > >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > >> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >> > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Stephen Mallette <sp...@gmail.com>]

Hi Ahmad, thanks for the updates! I've spent some time looking at the
changes and have added some comments and questions. I've pointed out a
couple things for future work that I think will be important, but nothing
that needs to be changed for this PR in my mind. Looking forward to seeing
your responses, other community feedback, and ultimately a merge of the
provider - take care!

On Thu, Mar 6, 2025 at 7:28 AM Ahmad Farhan <ah...@gmail.com>
wrote:

> Hello again!
> After several days of debugging and branch restructuring I managed to
> create another PR https://github.com/apache/airflow/pull/47446, this is
> due
> to major merge conflicts after the www clean up.
> The renaming of the provider to apache-tinkerpop is done but I kept the
> integration testing as 'gremlin' to avoid the conflict/confusion with the
> server name within the CI.
>
> Please review the PR thoroughly and I am happy to work on the following
> steps.
>
> Thanks,
> Farhan
>
> On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <ah...@gmail.com>
> wrote:
>
> > I did look into the naming and I thought that it would need to be
> > discussed at some point before the lazy consensus stage after the dev
> work
> > is done but I guess I was wrong :)
> >
> > I read through some docs regarding the naming and I kept thinking that
> > Apache Gremlin might not be right, so I decided to remove 'Apache' from
> all
> > doc strings. One thing that popped in one of the documentations from
> > Microsoft that says Apache Gremlin (here
> > https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction)
> > which confused me.
> >
> > I will change the folder to apache/tinkerpop. Thanks Stephen for the
> > in-depth explanation.
> >
> > On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> Cool. I will let Ahmad comment, but I think we found the **someone** who
> >> will help in case there are some future issues with the
> Tinkerpop/Gremlin
> >> provider.
> >> While I like Gremlin better (it's just a cool name and I like the logo,
> >> tinkerpop has a cool logo as well
> https://tinkerpop.apache.org/index.html
> >> ).
> >>
> >> So as long as we decide not to use common.graph -> I am fine with both
> :)
> >>
> >> j.
> >>
> >>
> >> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <sp...@gmail.com>
> >> wrote:
> >>
> >> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> >> >
> >> > > > In the interest of ASF trademarks, I would suggest it be called
> >> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and
> >> the
> >> > > like, as it is now with GremlinOperator. I think this makes sense
> >> because
> >> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> >> > > similarly suggest that references to "Apache Gremlin" and the like
> >> become
> >> > > "Apache TinkerPop".
> >> > >
> >> > > That's an interesting one - indeed TinkerPop is the PMC/ Framework -
> >> > > Gremlin is the language.
> >> > >
> >> > > I am not sure we are actually using TinkerPop here - because
> >> TinkerPop is
> >> > > the whole framework - Ahmad, can you explain the relation there -
> are
> >> > those
> >> > > other systems simply implement Gremlin as language or do they use
> >> > TinkerPop
> >> > > for something / as a backend?
> >> > >
> >> >
> >> > I'm sure Ahmad could answer but I'll quickly offer my take. I think
> >> that in
> >> > this case we should prefer "TinkerPop" over Gremlin as a top-level
> name
> >> > particularly because it's prefixed with "Apache" and there is no
> "Apache
> >> > Gremlin" which I tend to think is confusing when the words are that
> >> close
> >> > together. I can't recall over the years just how many times I've asked
> >> for
> >> > corrections in blog posts. :)
> >> >
> >> > Because that's a bit of a conceptual difference here. For example in
> the
> >> > > provider we are importing https://pypi.org/project/gremlinpython
> not
> >> > > "tinkerpop" - and it also does not have tinkerpop as dependency.
> >> > >
> >> >
> >> > A bit of history goes along with a lot of our naming for what we term
> >> > Gremlin Language Variants (GLVs), like gremlinpython, which are
> >> variants of
> >> > Gremlin natively implemented to allow users to express Gremlin in the
> >> > idioms of their own language. They also provide driver connectivity to
> >> > compatible servers. TinkerPop has mostly inherited all of its language
> >> > variants, including gremlinpython which was the first, from
> third-party
> >> > community developers. As a project, we didn't really get a hand in the
> >> > naming so with those projects already in heavy use we just kinda of
> >> stuck
> >> > to it and even doubled-down (like when we built gremlin-go within the
> >> ASF).
> >> >
> >> > I think in this case, your project organization under "apache" seems
> to
> >> > almost lend itself nicely to apache/tinkerpop. i think users will
> >> recognize
> >> > it as equally as they recognize Gremlin.
> >> >
> >> >
> >> > >
> >> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we should
> >> ask
> >> > > tinkerpop PMC what they think about it?
> >> > >
> >> >
> >> > Gremlin is not an ASF trademark. That was debated for quite a long
> time
> >> > with trademarks@ along with deciding if Gremlin, the character and
> his
> >> > friends[1], were to be protected. In the end, for reasons I'm not
> sure I
> >> > quite remember, the ASF didn't think it was necessary.
> >> >
> >> > Anyway, I'm not sure if you noted my earlier post[2] but I'm one of
> the
> >> > original contributors to Apache TinkerPop, even before we brought it
> to
> >> the
> >> > ASF so I'm pretty familiar with our project.  :)
> >> >
> >> > [1]
> >> >
> >> >
> >>
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> >> > [2] https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> >> >
> >> >
> >> > >
> >> > >
> >> > > J.
> >> > >
> >> > >
> >> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> >> spmallette@apache.org>
> >> > > wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> >> > > > > Yeah . `apache/gremlin" seems like a better option then. Does
> >> anyone
> >> > > have
> >> > > > > anything against it?
> >> > > >
> >> > > > In the interest of ASF trademarks, I would suggest it be called
> >> > > > "apache/tinkerpop" with "Gremlin" naming reserved for operators
> and
> >> the
> >> > > > like, as it is now with GremlinOperator. I think this makes sense
> >> > because
> >> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> >> > > > similarly suggest that references to "Apache Gremlin" and the like
> >> > become
> >> > > > "Apache TinkerPop".
> >> > > >
> >> > > > > I think we are pretty happy with accepting "other
> >> > > > > apache" projects as providers, so I see no issue with Gremlin -
> >> > knowing
> >> > > > > that we can always reach out to our friendly Apache Community in
> >> case
> >> > > of
> >> > > > > any issues. So - unless we do not hear any "opposition" in a few
> >> > days,
> >> > > I
> >> > > > > think it would make sense if you start `[LAZY CONSENSUS]`
> thread -
> >> > > > > without a need for `[VOTE]` thread.
> >> > > > >
> >> > > > > One thing though that I would love to have - is to also have an
> >> > > > integration
> >> > > > > test if possible (we had it with apache.kafka for example) -
> those
> >> > are
> >> > > > > tests that could run **some** graphdb database locally (via
> >> > > > docker-compose)
> >> > > > > and run a very rudimentary checks against a "real" database,
> not a
> >> > > mocked
> >> > > > > call. That would make it more robust.
> >> > > > >
> >> > > > > More about integration tests, how to build, run, test them and
> >> > > integrate
> >> > > > > them in our CI can be found here:
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> >> > > > > - happy to help if you are stuck with it.
> >> > > > >
> >> > > > > J.
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> >> > > ahmad.farhan9715@gmail.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > I pushed changes to move the provider into the “apache”
> >> directory.
> >> > > > After
> >> > > > > > updating the class references across the project, I re-tested
> >> and
> >> > all
> >> > > > tests
> >> > > > > > passed.
> >> > > > > >
> >> > > > > > Regarding the use of Gremlin (or another graph query language
> >> like
> >> > > > Cypher
> >> > > > > > and SPARQL) for a common package approach, here are my
> thoughts
> >> on
> >> > > the
> >> > > > pros
> >> > > > > > and cons:
> >> > > > > >
> >> > > > > > pros (I can see only one):
> >> > > > > >
> >> > > > > >    - Gremlin has been widely adopted by different cloud
> vendors
> >> > (e.g.
> >> > > > Azure
> >> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as
> in
> >> > > > self-hosted
> >> > > > > >    environments.
> >> > > > > >
> >> > > > > > cons:
> >> > > > > >
> >> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have
> >> their
> >> > > own
> >> > > > > >    drivers for executing queries.
> >> > > > > >    - To achieve a common abstraction, a wrapper around each
> >> driver
> >> > > > would be
> >> > > > > >    required. Each driver has its own connection parameters,
> >> > > underlying
> >> > > > > >    protocols, and may need method overrides for compatibility
> >> with
> >> > > > > > different
> >> > > > > >    Python versions.
> >> > > > > >    - Not all vendors support every query language; for
> instance,
> >> > > > Gremlin
> >> > > > > >    for Neo4j has been deprecated in recent releases, while
> >> Cosmos
> >> > DB
> >> > > > does
> >> > > > > > not
> >> > > > > >    support Cypher or SPARQL.
> >> > > > > >
> >> > > > > > While it would be ideal to have a unified graph query language
> >> and
> >> > > > driver
> >> > > > > > that works seamlessly across different vendors, such a
> solution
> >> > does
> >> > > > not
> >> > > > > > exist at the moment. In my opinion, implementing
> >> provider-specific
> >> > > > > > solutions for each query language (Gremlin, Cypher, SPARQL) is
> >> more
> >> > > > > > realistic and practical given the current landscape.
> >> > > > > >
> >> > > > > > Happy to discuss further or answer any questions!
> >> > > > > >
> >> > > > > > Farhan
> >> > > > > >
> >> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> >> > > > ahmad.farhan9715@gmail.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > I have worked with two different graph database
> vendors—Azure
> >> > > Cosmos
> >> > > > DB
> >> > > > > > > and Neo4j. During our migration to Neo4j, we discovered that
> >> > using
> >> > > > the
> >> > > > > > > Gremlin language wasn’t possible; we were forced to rewrite
> >> all
> >> > our
> >> > > > > > queries
> >> > > > > > > into Cypher, which is the native language for Neo4j and, in
> my
> >> > > > > > experience,
> >> > > > > > > much simpler for querying.
> >> > > > > > >
> >> > > > > > > This situation highlights a key challenge for a common
> >> > abstraction:
> >> > > > the
> >> > > > > > > underlying query languages and connection/authentication
> >> > mechanisms
> >> > > > vary
> >> > > > > > > significantly. Gremlin is not only different from Cypher in
> >> > syntax
> >> > > > but is
> >> > > > > > > also deprecated for Neo4j (see
> >> > > > > > >
> >> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> >> > ).
> >> > > > > > >
> >> > > > > > > The question would be how can the common approach
> accommodate
> >> > these
> >> > > > > > > different query languages?
> >> > > > > > >
> >> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
> >> jarek@potiuk.com>
> >> > > > wrote:
> >> > > > > > >
> >> > > > > > >> Without deep looking at the code I love the idea - it's
> very
> >> > > > similar to
> >> > > > > > >> what we have for common.sql and common.io - and soon
> >> > > > common.messaging
> >> > > > > > - I
> >> > > > > > >> also - long time ago - suggested common.dataframe that
> >> someone
> >> > > could
> >> > > > > > >> submit
> >> > > > > > >> using Apache Ibis:
> >> > > > > > >>
> >> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> >> > > -
> >> > > > > > >> similarly I believe there was an idea about common.llm ...
> >> > > > > > >>
> >> > > > > > >> I think the "common" pattern is a great one for Airflow, to
> >> > build
> >> > > > on top
> >> > > > > > >> of
> >> > > > > > >> "other giants" who build those common abstractions that you
> >> can
> >> > > > easily
> >> > > > > > >> switch between different implementations of various data
> >> access
> >> > > > layers.
> >> > > > > > >>
> >> > > > > > >> My suggestion and question - would be however (not very
> >> strong
> >> > on
> >> > > > it, I
> >> > > > > > >> would love to hear what others think, I know it's been
> >> somewhat
> >> > > > > > >> contentious
> >> > > > > > >> when I started the ibis discussion) - would be to make it
> >> > > > > > "common.graph",
> >> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> >> > "apache.ibis"
> >> > > -
> >> > > > just
> >> > > > > > >> to
> >> > > > > > >> stress that those are not implementations of particular
> >> service
> >> > > but
> >> > > > > > >> opinionated choice of particular technology to do "common"
> >> > > > operations.
> >> > > > > > >> This
> >> > > > > > >> is what essentially "common.io" is . - it should be named
> >> > > "fsspec"
> >> > > > > > >> provider
> >> > > > > > >> if we were to name it by the "library" that implemented it.
> >> > > > > > >>
> >> > > > > > >> J.
> >> > > > > > >>
> >> > > > > > >>
> >> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> >> > > > > > ahmad.farhan9715@gmail.com>
> >> > > > > > >> wrote:
> >> > > > > > >>
> >> > > > > > >> > Hi Everyone,
> >> > > > > > >> >
> >> > > > > > >> > I’ve created a draft PR (
> >> > > > https://github.com/apache/airflow/pull/46977
> >> > > > > > )
> >> > > > > > >> to
> >> > > > > > >> > introduce and discuss a new provider for using
> Gremlin—the
> >> > graph
> >> > > > > > >> traversal
> >> > > > > > >> > language of Apache TinkerPop (more details here:
> >> > > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> >> > > supported
> >> > > > by
> >> > > > > > >> > various
> >> > > > > > >> > graph database vendors such as Azure Cosmos DB and Amazon
> >> > > Neptune.
> >> > > > > > >> > Previously, I had to develop a custom hook to query data
> >> from
> >> > > > Azure
> >> > > > > > >> Cosmos
> >> > > > > > >> > DB using Apache Gremlin.
> >> > > > > > >> >
> >> > > > > > >> > I managed to create a provider and run it locally on the
> >> main
> >> > > > branch.
> >> > > > > > >> > However, I ran into the BaseHook issue (
> >> > > > > > >> > https://github.com/apache/airflow/issues/45233) on that
> >> > branch,
> >> > > > so I
> >> > > > > > >> ended
> >> > > > > > >> > up testing it fully on the v2-10-test branch. The PR
> >> should be
> >> > > > > > complete,
> >> > > > > > >> > but I’ve kept it as a draft for now while we discuss the
> >> > > provider.
> >> > > > > > >> >
> >> > > > > > >> > I’m a new contributor, so I’m especially eager to hear
> your
> >> > > > feedback.
> >> > > > > > >> > Comments on the PR is very welcome, and please feel free
> to
> >> > > reach
> >> > > > out
> >> > > > > > >> with
> >> > > > > > >> > any questions via email or Slack.
> >> > > > > > >> >
> >> > > > > > >> > Thanks,
> >> > > > > > >> > Ahmad Farhan
> >> > > > > > >> >
> >> > > > > > >>
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> >> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

Hello again!
After several days of debugging and branch restructuring I managed to
create another PR https://github.com/apache/airflow/pull/47446, this is due
to major merge conflicts after the www clean up.
The renaming of the provider to apache-tinkerpop is done but I kept the
integration testing as 'gremlin' to avoid the conflict/confusion with the
server name within the CI.

Please review the PR thoroughly and I am happy to work on the following
steps.

Thanks,
Farhan

On Wed, Feb 26, 2025 at 7:43 PM Ahmad Farhan <ah...@gmail.com>
wrote:

> I did look into the naming and I thought that it would need to be
> discussed at some point before the lazy consensus stage after the dev work
> is done but I guess I was wrong :)
>
> I read through some docs regarding the naming and I kept thinking that
> Apache Gremlin might not be right, so I decided to remove 'Apache' from all
> doc strings. One thing that popped in one of the documentations from
> Microsoft that says Apache Gremlin (here
> https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction)
> which confused me.
>
> I will change the folder to apache/tinkerpop. Thanks Stephen for the
> in-depth explanation.
>
> On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Cool. I will let Ahmad comment, but I think we found the **someone** who
>> will help in case there are some future issues with the Tinkerpop/Gremlin
>> provider.
>> While I like Gremlin better (it's just a cool name and I like the logo,
>> tinkerpop has a cool logo as well https://tinkerpop.apache.org/index.html
>> ).
>>
>> So as long as we decide not to use common.graph -> I am fine with both :)
>>
>> j.
>>
>>
>> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <sp...@gmail.com>
>> wrote:
>>
>> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>> >
>> > > > In the interest of ASF trademarks, I would suggest it be called
>> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and
>> the
>> > > like, as it is now with GremlinOperator. I think this makes sense
>> because
>> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
>> > > similarly suggest that references to "Apache Gremlin" and the like
>> become
>> > > "Apache TinkerPop".
>> > >
>> > > That's an interesting one - indeed TinkerPop is the PMC/ Framework -
>> > > Gremlin is the language.
>> > >
>> > > I am not sure we are actually using TinkerPop here - because
>> TinkerPop is
>> > > the whole framework - Ahmad, can you explain the relation there - are
>> > those
>> > > other systems simply implement Gremlin as language or do they use
>> > TinkerPop
>> > > for something / as a backend?
>> > >
>> >
>> > I'm sure Ahmad could answer but I'll quickly offer my take. I think
>> that in
>> > this case we should prefer "TinkerPop" over Gremlin as a top-level name
>> > particularly because it's prefixed with "Apache" and there is no "Apache
>> > Gremlin" which I tend to think is confusing when the words are that
>> close
>> > together. I can't recall over the years just how many times I've asked
>> for
>> > corrections in blog posts. :)
>> >
>> > Because that's a bit of a conceptual difference here. For example in the
>> > > provider we are importing https://pypi.org/project/gremlinpython not
>> > > "tinkerpop" - and it also does not have tinkerpop as dependency.
>> > >
>> >
>> > A bit of history goes along with a lot of our naming for what we term
>> > Gremlin Language Variants (GLVs), like gremlinpython, which are
>> variants of
>> > Gremlin natively implemented to allow users to express Gremlin in the
>> > idioms of their own language. They also provide driver connectivity to
>> > compatible servers. TinkerPop has mostly inherited all of its language
>> > variants, including gremlinpython which was the first, from third-party
>> > community developers. As a project, we didn't really get a hand in the
>> > naming so with those projects already in heavy use we just kinda of
>> stuck
>> > to it and even doubled-down (like when we built gremlin-go within the
>> ASF).
>> >
>> > I think in this case, your project organization under "apache" seems to
>> > almost lend itself nicely to apache/tinkerpop. i think users will
>> recognize
>> > it as equally as they recognize Gremlin.
>> >
>> >
>> > >
>> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we should
>> ask
>> > > tinkerpop PMC what they think about it?
>> > >
>> >
>> > Gremlin is not an ASF trademark. That was debated for quite a long time
>> > with trademarks@ along with deciding if Gremlin, the character and his
>> > friends[1], were to be protected. In the end, for reasons I'm not sure I
>> > quite remember, the ASF didn't think it was necessary.
>> >
>> > Anyway, I'm not sure if you noted my earlier post[2] but I'm one of the
>> > original contributors to Apache TinkerPop, even before we brought it to
>> the
>> > ASF so I'm pretty familiar with our project.  :)
>> >
>> > [1]
>> >
>> >
>> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
>> > [2] https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
>> >
>> >
>> > >
>> > >
>> > > J.
>> > >
>> > >
>> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
>> spmallette@apache.org>
>> > > wrote:
>> > >
>> > > >
>> > > >
>> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
>> > > > > Yeah . `apache/gremlin" seems like a better option then. Does
>> anyone
>> > > have
>> > > > > anything against it?
>> > > >
>> > > > In the interest of ASF trademarks, I would suggest it be called
>> > > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and
>> the
>> > > > like, as it is now with GremlinOperator. I think this makes sense
>> > because
>> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
>> > > > similarly suggest that references to "Apache Gremlin" and the like
>> > become
>> > > > "Apache TinkerPop".
>> > > >
>> > > > > I think we are pretty happy with accepting "other
>> > > > > apache" projects as providers, so I see no issue with Gremlin -
>> > knowing
>> > > > > that we can always reach out to our friendly Apache Community in
>> case
>> > > of
>> > > > > any issues. So - unless we do not hear any "opposition" in a few
>> > days,
>> > > I
>> > > > > think it would make sense if you start `[LAZY CONSENSUS]` thread -
>> > > > > without a need for `[VOTE]` thread.
>> > > > >
>> > > > > One thing though that I would love to have - is to also have an
>> > > > integration
>> > > > > test if possible (we had it with apache.kafka for example) - those
>> > are
>> > > > > tests that could run **some** graphdb database locally (via
>> > > > docker-compose)
>> > > > > and run a very rudimentary checks against a "real" database, not a
>> > > mocked
>> > > > > call. That would make it more robust.
>> > > > >
>> > > > > More about integration tests, how to build, run, test them and
>> > > integrate
>> > > > > them in our CI can be found here:
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
>> > > > > - happy to help if you are stuck with it.
>> > > > >
>> > > > > J.
>> > > > >
>> > > > >
>> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
>> > > ahmad.farhan9715@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I pushed changes to move the provider into the “apache”
>> directory.
>> > > > After
>> > > > > > updating the class references across the project, I re-tested
>> and
>> > all
>> > > > tests
>> > > > > > passed.
>> > > > > >
>> > > > > > Regarding the use of Gremlin (or another graph query language
>> like
>> > > > Cypher
>> > > > > > and SPARQL) for a common package approach, here are my thoughts
>> on
>> > > the
>> > > > pros
>> > > > > > and cons:
>> > > > > >
>> > > > > > pros (I can see only one):
>> > > > > >
>> > > > > >    - Gremlin has been widely adopted by different cloud vendors
>> > (e.g.
>> > > > Azure
>> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
>> > > > self-hosted
>> > > > > >    environments.
>> > > > > >
>> > > > > > cons:
>> > > > > >
>> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have
>> their
>> > > own
>> > > > > >    drivers for executing queries.
>> > > > > >    - To achieve a common abstraction, a wrapper around each
>> driver
>> > > > would be
>> > > > > >    required. Each driver has its own connection parameters,
>> > > underlying
>> > > > > >    protocols, and may need method overrides for compatibility
>> with
>> > > > > > different
>> > > > > >    Python versions.
>> > > > > >    - Not all vendors support every query language; for instance,
>> > > > Gremlin
>> > > > > >    for Neo4j has been deprecated in recent releases, while
>> Cosmos
>> > DB
>> > > > does
>> > > > > > not
>> > > > > >    support Cypher or SPARQL.
>> > > > > >
>> > > > > > While it would be ideal to have a unified graph query language
>> and
>> > > > driver
>> > > > > > that works seamlessly across different vendors, such a solution
>> > does
>> > > > not
>> > > > > > exist at the moment. In my opinion, implementing
>> provider-specific
>> > > > > > solutions for each query language (Gremlin, Cypher, SPARQL) is
>> more
>> > > > > > realistic and practical given the current landscape.
>> > > > > >
>> > > > > > Happy to discuss further or answer any questions!
>> > > > > >
>> > > > > > Farhan
>> > > > > >
>> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
>> > > > ahmad.farhan9715@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > I have worked with two different graph database vendors—Azure
>> > > Cosmos
>> > > > DB
>> > > > > > > and Neo4j. During our migration to Neo4j, we discovered that
>> > using
>> > > > the
>> > > > > > > Gremlin language wasn’t possible; we were forced to rewrite
>> all
>> > our
>> > > > > > queries
>> > > > > > > into Cypher, which is the native language for Neo4j and, in my
>> > > > > > experience,
>> > > > > > > much simpler for querying.
>> > > > > > >
>> > > > > > > This situation highlights a key challenge for a common
>> > abstraction:
>> > > > the
>> > > > > > > underlying query languages and connection/authentication
>> > mechanisms
>> > > > vary
>> > > > > > > significantly. Gremlin is not only different from Cypher in
>> > syntax
>> > > > but is
>> > > > > > > also deprecated for Neo4j (see
>> > > > > > >
>> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
>> > ).
>> > > > > > >
>> > > > > > > The question would be how can the common approach accommodate
>> > these
>> > > > > > > different query languages?
>> > > > > > >
>> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <
>> jarek@potiuk.com>
>> > > > wrote:
>> > > > > > >
>> > > > > > >> Without deep looking at the code I love the idea - it's very
>> > > > similar to
>> > > > > > >> what we have for common.sql and common.io - and soon
>> > > > common.messaging
>> > > > > > - I
>> > > > > > >> also - long time ago - suggested common.dataframe that
>> someone
>> > > could
>> > > > > > >> submit
>> > > > > > >> using Apache Ibis:
>> > > > > > >>
>> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
>> > > -
>> > > > > > >> similarly I believe there was an idea about common.llm ...
>> > > > > > >>
>> > > > > > >> I think the "common" pattern is a great one for Airflow, to
>> > build
>> > > > on top
>> > > > > > >> of
>> > > > > > >> "other giants" who build those common abstractions that you
>> can
>> > > > easily
>> > > > > > >> switch between different implementations of various data
>> access
>> > > > layers.
>> > > > > > >>
>> > > > > > >> My suggestion and question - would be however (not very
>> strong
>> > on
>> > > > it, I
>> > > > > > >> would love to hear what others think, I know it's been
>> somewhat
>> > > > > > >> contentious
>> > > > > > >> when I started the ibis discussion) - would be to make it
>> > > > > > "common.graph",
>> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
>> > "apache.ibis"
>> > > -
>> > > > just
>> > > > > > >> to
>> > > > > > >> stress that those are not implementations of particular
>> service
>> > > but
>> > > > > > >> opinionated choice of particular technology to do "common"
>> > > > operations.
>> > > > > > >> This
>> > > > > > >> is what essentially "common.io" is . - it should be named
>> > > "fsspec"
>> > > > > > >> provider
>> > > > > > >> if we were to name it by the "library" that implemented it.
>> > > > > > >>
>> > > > > > >> J.
>> > > > > > >>
>> > > > > > >>
>> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
>> > > > > > ahmad.farhan9715@gmail.com>
>> > > > > > >> wrote:
>> > > > > > >>
>> > > > > > >> > Hi Everyone,
>> > > > > > >> >
>> > > > > > >> > I’ve created a draft PR (
>> > > > https://github.com/apache/airflow/pull/46977
>> > > > > > )
>> > > > > > >> to
>> > > > > > >> > introduce and discuss a new provider for using Gremlin—the
>> > graph
>> > > > > > >> traversal
>> > > > > > >> > language of Apache TinkerPop (more details here:
>> > > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
>> > > supported
>> > > > by
>> > > > > > >> > various
>> > > > > > >> > graph database vendors such as Azure Cosmos DB and Amazon
>> > > Neptune.
>> > > > > > >> > Previously, I had to develop a custom hook to query data
>> from
>> > > > Azure
>> > > > > > >> Cosmos
>> > > > > > >> > DB using Apache Gremlin.
>> > > > > > >> >
>> > > > > > >> > I managed to create a provider and run it locally on the
>> main
>> > > > branch.
>> > > > > > >> > However, I ran into the BaseHook issue (
>> > > > > > >> > https://github.com/apache/airflow/issues/45233) on that
>> > branch,
>> > > > so I
>> > > > > > >> ended
>> > > > > > >> > up testing it fully on the v2-10-test branch. The PR
>> should be
>> > > > > > complete,
>> > > > > > >> > but I’ve kept it as a draft for now while we discuss the
>> > > provider.
>> > > > > > >> >
>> > > > > > >> > I’m a new contributor, so I’m especially eager to hear your
>> > > > feedback.
>> > > > > > >> > Comments on the PR is very welcome, and please feel free to
>> > > reach
>> > > > out
>> > > > > > >> with
>> > > > > > >> > any questions via email or Slack.
>> > > > > > >> >
>> > > > > > >> > Thanks,
>> > > > > > >> > Ahmad Farhan
>> > > > > > >> >
>> > > > > > >>
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> ---------------------------------------------------------------------
>> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
>> > > > For additional commands, e-mail: dev-help@airflow.apache.org
>> > > >
>> > > >
>> > >
>> >
>>
>

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

I did look into the naming and I thought that it would need to be discussed
at some point before the lazy consensus stage after the dev work is done
but I guess I was wrong :)

I read through some docs regarding the naming and I kept thinking that
Apache Gremlin might not be right, so I decided to remove 'Apache' from all
doc strings. One thing that popped in one of the documentations from
Microsoft that says Apache Gremlin (here
https://learn.microsoft.com/en-us/azure/cosmos-db/gremlin/introduction)
which confused me.

I will change the folder to apache/tinkerpop. Thanks Stephen for the
in-depth explanation.

On Wed, Feb 26, 2025 at 6:56 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Cool. I will let Ahmad comment, but I think we found the **someone** who
> will help in case there are some future issues with the Tinkerpop/Gremlin
> provider.
> While I like Gremlin better (it's just a cool name and I like the logo,
> tinkerpop has a cool logo as well https://tinkerpop.apache.org/index.html
> ).
>
> So as long as we decide not to use common.graph -> I am fine with both :)
>
> j.
>
>
> On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <sp...@gmail.com>
> wrote:
>
> > On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> > > > In the interest of ASF trademarks, I would suggest it be called
> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> > > like, as it is now with GremlinOperator. I think this makes sense
> because
> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > > similarly suggest that references to "Apache Gremlin" and the like
> become
> > > "Apache TinkerPop".
> > >
> > > That's an interesting one - indeed TinkerPop is the PMC/ Framework -
> > > Gremlin is the language.
> > >
> > > I am not sure we are actually using TinkerPop here - because TinkerPop
> is
> > > the whole framework - Ahmad, can you explain the relation there - are
> > those
> > > other systems simply implement Gremlin as language or do they use
> > TinkerPop
> > > for something / as a backend?
> > >
> >
> > I'm sure Ahmad could answer but I'll quickly offer my take. I think that
> in
> > this case we should prefer "TinkerPop" over Gremlin as a top-level name
> > particularly because it's prefixed with "Apache" and there is no "Apache
> > Gremlin" which I tend to think is confusing when the words are that close
> > together. I can't recall over the years just how many times I've asked
> for
> > corrections in blog posts. :)
> >
> > Because that's a bit of a conceptual difference here. For example in the
> > > provider we are importing https://pypi.org/project/gremlinpython not
> > > "tinkerpop" - and it also does not have tinkerpop as dependency.
> > >
> >
> > A bit of history goes along with a lot of our naming for what we term
> > Gremlin Language Variants (GLVs), like gremlinpython, which are variants
> of
> > Gremlin natively implemented to allow users to express Gremlin in the
> > idioms of their own language. They also provide driver connectivity to
> > compatible servers. TinkerPop has mostly inherited all of its language
> > variants, including gremlinpython which was the first, from third-party
> > community developers. As a project, we didn't really get a hand in the
> > naming so with those projects already in heavy use we just kinda of stuck
> > to it and even doubled-down (like when we built gremlin-go within the
> ASF).
> >
> > I think in this case, your project organization under "apache" seems to
> > almost lend itself nicely to apache/tinkerpop. i think users will
> recognize
> > it as equally as they recognize Gremlin.
> >
> >
> > >
> > > I wonder if Gremlin is also a Trademark by Apache ? Maybe we should ask
> > > tinkerpop PMC what they think about it?
> > >
> >
> > Gremlin is not an ASF trademark. That was debated for quite a long time
> > with trademarks@ along with deciding if Gremlin, the character and his
> > friends[1], were to be protected. In the end, for reasons I'm not sure I
> > quite remember, the ASF didn't think it was necessary.
> >
> > Anyway, I'm not sure if you noted my earlier post[2] but I'm one of the
> > original contributors to Apache TinkerPop, even before we brought it to
> the
> > ASF so I'm pretty familiar with our project.  :)
> >
> > [1]
> >
> >
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> > [2] https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
> >
> >
> > >
> > >
> > > J.
> > >
> > >
> > > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <
> spmallette@apache.org>
> > > wrote:
> > >
> > > >
> > > >
> > > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > > > Yeah . `apache/gremlin" seems like a better option then. Does
> anyone
> > > have
> > > > > anything against it?
> > > >
> > > > In the interest of ASF trademarks, I would suggest it be called
> > > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and
> the
> > > > like, as it is now with GremlinOperator. I think this makes sense
> > because
> > > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > > > similarly suggest that references to "Apache Gremlin" and the like
> > become
> > > > "Apache TinkerPop".
> > > >
> > > > > I think we are pretty happy with accepting "other
> > > > > apache" projects as providers, so I see no issue with Gremlin -
> > knowing
> > > > > that we can always reach out to our friendly Apache Community in
> case
> > > of
> > > > > any issues. So - unless we do not hear any "opposition" in a few
> > days,
> > > I
> > > > > think it would make sense if you start `[LAZY CONSENSUS]` thread -
> > > > > without a need for `[VOTE]` thread.
> > > > >
> > > > > One thing though that I would love to have - is to also have an
> > > > integration
> > > > > test if possible (we had it with apache.kafka for example) - those
> > are
> > > > > tests that could run **some** graphdb database locally (via
> > > > docker-compose)
> > > > > and run a very rudimentary checks against a "real" database, not a
> > > mocked
> > > > > call. That would make it more robust.
> > > > >
> > > > > More about integration tests, how to build, run, test them and
> > > integrate
> > > > > them in our CI can be found here:
> > > > >
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > > > - happy to help if you are stuck with it.
> > > > >
> > > > > J.
> > > > >
> > > > >
> > > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > > ahmad.farhan9715@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I pushed changes to move the provider into the “apache”
> directory.
> > > > After
> > > > > > updating the class references across the project, I re-tested and
> > all
> > > > tests
> > > > > > passed.
> > > > > >
> > > > > > Regarding the use of Gremlin (or another graph query language
> like
> > > > Cypher
> > > > > > and SPARQL) for a common package approach, here are my thoughts
> on
> > > the
> > > > pros
> > > > > > and cons:
> > > > > >
> > > > > > pros (I can see only one):
> > > > > >
> > > > > >    - Gremlin has been widely adopted by different cloud vendors
> > (e.g.
> > > > Azure
> > > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
> > > > self-hosted
> > > > > >    environments.
> > > > > >
> > > > > > cons:
> > > > > >
> > > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have
> their
> > > own
> > > > > >    drivers for executing queries.
> > > > > >    - To achieve a common abstraction, a wrapper around each
> driver
> > > > would be
> > > > > >    required. Each driver has its own connection parameters,
> > > underlying
> > > > > >    protocols, and may need method overrides for compatibility
> with
> > > > > > different
> > > > > >    Python versions.
> > > > > >    - Not all vendors support every query language; for instance,
> > > > Gremlin
> > > > > >    for Neo4j has been deprecated in recent releases, while Cosmos
> > DB
> > > > does
> > > > > > not
> > > > > >    support Cypher or SPARQL.
> > > > > >
> > > > > > While it would be ideal to have a unified graph query language
> and
> > > > driver
> > > > > > that works seamlessly across different vendors, such a solution
> > does
> > > > not
> > > > > > exist at the moment. In my opinion, implementing
> provider-specific
> > > > > > solutions for each query language (Gremlin, Cypher, SPARQL) is
> more
> > > > > > realistic and practical given the current landscape.
> > > > > >
> > > > > > Happy to discuss further or answer any questions!
> > > > > >
> > > > > > Farhan
> > > > > >
> > > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > > > ahmad.farhan9715@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I have worked with two different graph database vendors—Azure
> > > Cosmos
> > > > DB
> > > > > > > and Neo4j. During our migration to Neo4j, we discovered that
> > using
> > > > the
> > > > > > > Gremlin language wasn’t possible; we were forced to rewrite all
> > our
> > > > > > queries
> > > > > > > into Cypher, which is the native language for Neo4j and, in my
> > > > > > experience,
> > > > > > > much simpler for querying.
> > > > > > >
> > > > > > > This situation highlights a key challenge for a common
> > abstraction:
> > > > the
> > > > > > > underlying query languages and connection/authentication
> > mechanisms
> > > > vary
> > > > > > > significantly. Gremlin is not only different from Cypher in
> > syntax
> > > > but is
> > > > > > > also deprecated for Neo4j (see
> > > > > > >
> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> > ).
> > > > > > >
> > > > > > > The question would be how can the common approach accommodate
> > these
> > > > > > > different query languages?
> > > > > > >
> > > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <jarek@potiuk.com
> >
> > > > wrote:
> > > > > > >
> > > > > > >> Without deep looking at the code I love the idea - it's very
> > > > similar to
> > > > > > >> what we have for common.sql and common.io - and soon
> > > > common.messaging
> > > > > > - I
> > > > > > >> also - long time ago - suggested common.dataframe that someone
> > > could
> > > > > > >> submit
> > > > > > >> using Apache Ibis:
> > > > > > >>
> > https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > > -
> > > > > > >> similarly I believe there was an idea about common.llm ...
> > > > > > >>
> > > > > > >> I think the "common" pattern is a great one for Airflow, to
> > build
> > > > on top
> > > > > > >> of
> > > > > > >> "other giants" who build those common abstractions that you
> can
> > > > easily
> > > > > > >> switch between different implementations of various data
> access
> > > > layers.
> > > > > > >>
> > > > > > >> My suggestion and question - would be however (not very strong
> > on
> > > > it, I
> > > > > > >> would love to hear what others think, I know it's been
> somewhat
> > > > > > >> contentious
> > > > > > >> when I started the ibis discussion) - would be to make it
> > > > > > "common.graph",
> > > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> > "apache.ibis"
> > > -
> > > > just
> > > > > > >> to
> > > > > > >> stress that those are not implementations of particular
> service
> > > but
> > > > > > >> opinionated choice of particular technology to do "common"
> > > > operations.
> > > > > > >> This
> > > > > > >> is what essentially "common.io" is . - it should be named
> > > "fsspec"
> > > > > > >> provider
> > > > > > >> if we were to name it by the "library" that implemented it.
> > > > > > >>
> > > > > > >> J.
> > > > > > >>
> > > > > > >>
> > > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > > > > ahmad.farhan9715@gmail.com>
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Everyone,
> > > > > > >> >
> > > > > > >> > I’ve created a draft PR (
> > > > https://github.com/apache/airflow/pull/46977
> > > > > > )
> > > > > > >> to
> > > > > > >> > introduce and discuss a new provider for using Gremlin—the
> > graph
> > > > > > >> traversal
> > > > > > >> > language of Apache TinkerPop (more details here:
> > > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> > > supported
> > > > by
> > > > > > >> > various
> > > > > > >> > graph database vendors such as Azure Cosmos DB and Amazon
> > > Neptune.
> > > > > > >> > Previously, I had to develop a custom hook to query data
> from
> > > > Azure
> > > > > > >> Cosmos
> > > > > > >> > DB using Apache Gremlin.
> > > > > > >> >
> > > > > > >> > I managed to create a provider and run it locally on the
> main
> > > > branch.
> > > > > > >> > However, I ran into the BaseHook issue (
> > > > > > >> > https://github.com/apache/airflow/issues/45233) on that
> > branch,
> > > > so I
> > > > > > >> ended
> > > > > > >> > up testing it fully on the v2-10-test branch. The PR should
> be
> > > > > > complete,
> > > > > > >> > but I’ve kept it as a draft for now while we discuss the
> > > provider.
> > > > > > >> >
> > > > > > >> > I’m a new contributor, so I’m especially eager to hear your
> > > > feedback.
> > > > > > >> > Comments on the PR is very welcome, and please feel free to
> > > reach
> > > > out
> > > > > > >> with
> > > > > > >> > any questions via email or Slack.
> > > > > > >> >
> > > > > > >> > Thanks,
> > > > > > >> > Ahmad Farhan
> > > > > > >> >
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > > For additional commands, e-mail: dev-help@airflow.apache.org
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Jarek Potiuk <ja...@potiuk.com>]

Cool. I will let Ahmad comment, but I think we found the **someone** who
will help in case there are some future issues with the Tinkerpop/Gremlin
provider.
While I like Gremlin better (it's just a cool name and I like the logo,
tinkerpop has a cool logo as well https://tinkerpop.apache.org/index.html
).

So as long as we decide not to use common.graph -> I am fine with both :)

j.


On Wed, Feb 26, 2025 at 7:42 PM Stephen Mallette <sp...@gmail.com>
wrote:

> On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
> > > In the interest of ASF trademarks, I would suggest it be called
> > "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> > like, as it is now with GremlinOperator. I think this makes sense because
> > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > similarly suggest that references to "Apache Gremlin" and the like become
> > "Apache TinkerPop".
> >
> > That's an interesting one - indeed TinkerPop is the PMC/ Framework -
> > Gremlin is the language.
> >
> > I am not sure we are actually using TinkerPop here - because TinkerPop is
> > the whole framework - Ahmad, can you explain the relation there - are
> those
> > other systems simply implement Gremlin as language or do they use
> TinkerPop
> > for something / as a backend?
> >
>
> I'm sure Ahmad could answer but I'll quickly offer my take. I think that in
> this case we should prefer "TinkerPop" over Gremlin as a top-level name
> particularly because it's prefixed with "Apache" and there is no "Apache
> Gremlin" which I tend to think is confusing when the words are that close
> together. I can't recall over the years just how many times I've asked for
> corrections in blog posts. :)
>
> Because that's a bit of a conceptual difference here. For example in the
> > provider we are importing https://pypi.org/project/gremlinpython not
> > "tinkerpop" - and it also does not have tinkerpop as dependency.
> >
>
> A bit of history goes along with a lot of our naming for what we term
> Gremlin Language Variants (GLVs), like gremlinpython, which are variants of
> Gremlin natively implemented to allow users to express Gremlin in the
> idioms of their own language. They also provide driver connectivity to
> compatible servers. TinkerPop has mostly inherited all of its language
> variants, including gremlinpython which was the first, from third-party
> community developers. As a project, we didn't really get a hand in the
> naming so with those projects already in heavy use we just kinda of stuck
> to it and even doubled-down (like when we built gremlin-go within the ASF).
>
> I think in this case, your project organization under "apache" seems to
> almost lend itself nicely to apache/tinkerpop. i think users will recognize
> it as equally as they recognize Gremlin.
>
>
> >
> > I wonder if Gremlin is also a Trademark by Apache ? Maybe we should ask
> > tinkerpop PMC what they think about it?
> >
>
> Gremlin is not an ASF trademark. That was debated for quite a long time
> with trademarks@ along with deciding if Gremlin, the character and his
> friends[1], were to be protected. In the end, for reasons I'm not sure I
> quite remember, the ASF didn't think it was necessary.
>
> Anyway, I'm not sure if you noted my earlier post[2] but I'm one of the
> original contributors to Apache TinkerPop, even before we brought it to the
> ASF so I'm pretty familiar with our project.  :)
>
> [1]
>
> https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
> [2] https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y
>
>
> >
> >
> > J.
> >
> >
> > On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <sp...@apache.org>
> > wrote:
> >
> > >
> > >
> > > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > > Yeah . `apache/gremlin" seems like a better option then. Does anyone
> > have
> > > > anything against it?
> > >
> > > In the interest of ASF trademarks, I would suggest it be called
> > > "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> > > like, as it is now with GremlinOperator. I think this makes sense
> because
> > > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > > similarly suggest that references to "Apache Gremlin" and the like
> become
> > > "Apache TinkerPop".
> > >
> > > > I think we are pretty happy with accepting "other
> > > > apache" projects as providers, so I see no issue with Gremlin -
> knowing
> > > > that we can always reach out to our friendly Apache Community in case
> > of
> > > > any issues. So - unless we do not hear any "opposition" in a few
> days,
> > I
> > > > think it would make sense if you start `[LAZY CONSENSUS]` thread -
> > > > without a need for `[VOTE]` thread.
> > > >
> > > > One thing though that I would love to have - is to also have an
> > > integration
> > > > test if possible (we had it with apache.kafka for example) - those
> are
> > > > tests that could run **some** graphdb database locally (via
> > > docker-compose)
> > > > and run a very rudimentary checks against a "real" database, not a
> > mocked
> > > > call. That would make it more robust.
> > > >
> > > > More about integration tests, how to build, run, test them and
> > integrate
> > > > them in our CI can be found here:
> > > >
> > >
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > > - happy to help if you are stuck with it.
> > > >
> > > > J.
> > > >
> > > >
> > > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > I pushed changes to move the provider into the “apache” directory.
> > > After
> > > > > updating the class references across the project, I re-tested and
> all
> > > tests
> > > > > passed.
> > > > >
> > > > > Regarding the use of Gremlin (or another graph query language like
> > > Cypher
> > > > > and SPARQL) for a common package approach, here are my thoughts on
> > the
> > > pros
> > > > > and cons:
> > > > >
> > > > > pros (I can see only one):
> > > > >
> > > > >    - Gremlin has been widely adopted by different cloud vendors
> (e.g.
> > > Azure
> > > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
> > > self-hosted
> > > > >    environments.
> > > > >
> > > > > cons:
> > > > >
> > > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their
> > own
> > > > >    drivers for executing queries.
> > > > >    - To achieve a common abstraction, a wrapper around each driver
> > > would be
> > > > >    required. Each driver has its own connection parameters,
> > underlying
> > > > >    protocols, and may need method overrides for compatibility with
> > > > > different
> > > > >    Python versions.
> > > > >    - Not all vendors support every query language; for instance,
> > > Gremlin
> > > > >    for Neo4j has been deprecated in recent releases, while Cosmos
> DB
> > > does
> > > > > not
> > > > >    support Cypher or SPARQL.
> > > > >
> > > > > While it would be ideal to have a unified graph query language and
> > > driver
> > > > > that works seamlessly across different vendors, such a solution
> does
> > > not
> > > > > exist at the moment. In my opinion, implementing provider-specific
> > > > > solutions for each query language (Gremlin, Cypher, SPARQL) is more
> > > > > realistic and practical given the current landscape.
> > > > >
> > > > > Happy to discuss further or answer any questions!
> > > > >
> > > > > Farhan
> > > > >
> > > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > > ahmad.farhan9715@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I have worked with two different graph database vendors—Azure
> > Cosmos
> > > DB
> > > > > > and Neo4j. During our migration to Neo4j, we discovered that
> using
> > > the
> > > > > > Gremlin language wasn’t possible; we were forced to rewrite all
> our
> > > > > queries
> > > > > > into Cypher, which is the native language for Neo4j and, in my
> > > > > experience,
> > > > > > much simpler for querying.
> > > > > >
> > > > > > This situation highlights a key challenge for a common
> abstraction:
> > > the
> > > > > > underlying query languages and connection/authentication
> mechanisms
> > > vary
> > > > > > significantly. Gremlin is not only different from Cypher in
> syntax
> > > but is
> > > > > > also deprecated for Neo4j (see
> > > > > > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin
> ).
> > > > > >
> > > > > > The question would be how can the common approach accommodate
> these
> > > > > > different query languages?
> > > > > >
> > > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com>
> > > wrote:
> > > > > >
> > > > > >> Without deep looking at the code I love the idea - it's very
> > > similar to
> > > > > >> what we have for common.sql and common.io - and soon
> > > common.messaging
> > > > > - I
> > > > > >> also - long time ago - suggested common.dataframe that someone
> > could
> > > > > >> submit
> > > > > >> using Apache Ibis:
> > > > > >>
> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> > -
> > > > > >> similarly I believe there was an idea about common.llm ...
> > > > > >>
> > > > > >> I think the "common" pattern is a great one for Airflow, to
> build
> > > on top
> > > > > >> of
> > > > > >> "other giants" who build those common abstractions that you can
> > > easily
> > > > > >> switch between different implementations of various data access
> > > layers.
> > > > > >>
> > > > > >> My suggestion and question - would be however (not very strong
> on
> > > it, I
> > > > > >> would love to hear what others think, I know it's been somewhat
> > > > > >> contentious
> > > > > >> when I started the ibis discussion) - would be to make it
> > > > > "common.graph",
> > > > > >> "common.dataframe" - instead of "apache.gremlin" or
> "apache.ibis"
> > -
> > > just
> > > > > >> to
> > > > > >> stress that those are not implementations of particular service
> > but
> > > > > >> opinionated choice of particular technology to do "common"
> > > operations.
> > > > > >> This
> > > > > >> is what essentially "common.io" is . - it should be named
> > "fsspec"
> > > > > >> provider
> > > > > >> if we were to name it by the "library" that implemented it.
> > > > > >>
> > > > > >> J.
> > > > > >>
> > > > > >>
> > > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > > > ahmad.farhan9715@gmail.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Hi Everyone,
> > > > > >> >
> > > > > >> > I’ve created a draft PR (
> > > https://github.com/apache/airflow/pull/46977
> > > > > )
> > > > > >> to
> > > > > >> > introduce and discuss a new provider for using Gremlin—the
> graph
> > > > > >> traversal
> > > > > >> > language of Apache TinkerPop (more details here:
> > > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> > supported
> > > by
> > > > > >> > various
> > > > > >> > graph database vendors such as Azure Cosmos DB and Amazon
> > Neptune.
> > > > > >> > Previously, I had to develop a custom hook to query data from
> > > Azure
> > > > > >> Cosmos
> > > > > >> > DB using Apache Gremlin.
> > > > > >> >
> > > > > >> > I managed to create a provider and run it locally on the main
> > > branch.
> > > > > >> > However, I ran into the BaseHook issue (
> > > > > >> > https://github.com/apache/airflow/issues/45233) on that
> branch,
> > > so I
> > > > > >> ended
> > > > > >> > up testing it fully on the v2-10-test branch. The PR should be
> > > > > complete,
> > > > > >> > but I’ve kept it as a draft for now while we discuss the
> > provider.
> > > > > >> >
> > > > > >> > I’m a new contributor, so I’m especially eager to hear your
> > > feedback.
> > > > > >> > Comments on the PR is very welcome, and please feel free to
> > reach
> > > out
> > > > > >> with
> > > > > >> > any questions via email or Slack.
> > > > > >> >
> > > > > >> > Thanks,
> > > > > >> > Ahmad Farhan
> > > > > >> >
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > > For additional commands, e-mail: dev-help@airflow.apache.org
> > >
> > >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Stephen Mallette <sp...@gmail.com>]

On Wed, Feb 26, 2025 at 12:57 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> > In the interest of ASF trademarks, I would suggest it be called
> "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> like, as it is now with GremlinOperator. I think this makes sense because
> it is connecting to TinkerPop-enabled systems via Gremlin. I would
> similarly suggest that references to "Apache Gremlin" and the like become
> "Apache TinkerPop".
>
> That's an interesting one - indeed TinkerPop is the PMC/ Framework -
> Gremlin is the language.
>
> I am not sure we are actually using TinkerPop here - because TinkerPop is
> the whole framework - Ahmad, can you explain the relation there - are those
> other systems simply implement Gremlin as language or do they use TinkerPop
> for something / as a backend?
>

I'm sure Ahmad could answer but I'll quickly offer my take. I think that in
this case we should prefer "TinkerPop" over Gremlin as a top-level name
particularly because it's prefixed with "Apache" and there is no "Apache
Gremlin" which I tend to think is confusing when the words are that close
together. I can't recall over the years just how many times I've asked for
corrections in blog posts. :)

Because that's a bit of a conceptual difference here. For example in the
> provider we are importing https://pypi.org/project/gremlinpython not
> "tinkerpop" - and it also does not have tinkerpop as dependency.
>

A bit of history goes along with a lot of our naming for what we term
Gremlin Language Variants (GLVs), like gremlinpython, which are variants of
Gremlin natively implemented to allow users to express Gremlin in the
idioms of their own language. They also provide driver connectivity to
compatible servers. TinkerPop has mostly inherited all of its language
variants, including gremlinpython which was the first, from third-party
community developers. As a project, we didn't really get a hand in the
naming so with those projects already in heavy use we just kinda of stuck
to it and even doubled-down (like when we built gremlin-go within the ASF).

I think in this case, your project organization under "apache" seems to
almost lend itself nicely to apache/tinkerpop. i think users will recognize
it as equally as they recognize Gremlin.


>
> I wonder if Gremlin is also a Trademark by Apache ? Maybe we should ask
> tinkerpop PMC what they think about it?
>

Gremlin is not an ASF trademark. That was debated for quite a long time
with trademarks@ along with deciding if Gremlin, the character and his
friends[1], were to be protected. In the end, for reasons I'm not sure I
quite remember, the ASF didn't think it was necessary.

Anyway, I'm not sure if you noted my earlier post[2] but I'm one of the
original contributors to Apache TinkerPop, even before we brought it to the
ASF so I'm pretty familiar with our project.  :)

[1]
https://github.com/apache/tinkerpop/blob/master/docs/static/images/tinkerpop3-splash.png
[2] https://lists.apache.org/thread/9hf4t8hyk944fyo4q3nygczyo5xhk18y


>
>
> J.
>
>
> On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <sp...@apache.org>
> wrote:
>
> >
> >
> > On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > > Yeah . `apache/gremlin" seems like a better option then. Does anyone
> have
> > > anything against it?
> >
> > In the interest of ASF trademarks, I would suggest it be called
> > "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> > like, as it is now with GremlinOperator. I think this makes sense because
> > it is connecting to TinkerPop-enabled systems via Gremlin. I would
> > similarly suggest that references to "Apache Gremlin" and the like become
> > "Apache TinkerPop".
> >
> > > I think we are pretty happy with accepting "other
> > > apache" projects as providers, so I see no issue with Gremlin - knowing
> > > that we can always reach out to our friendly Apache Community in case
> of
> > > any issues. So - unless we do not hear any "opposition" in a few days,
> I
> > > think it would make sense if you start `[LAZY CONSENSUS]` thread -
> > > without a need for `[VOTE]` thread.
> > >
> > > One thing though that I would love to have - is to also have an
> > integration
> > > test if possible (we had it with apache.kafka for example) - those are
> > > tests that could run **some** graphdb database locally (via
> > docker-compose)
> > > and run a very rudimentary checks against a "real" database, not a
> mocked
> > > call. That would make it more robust.
> > >
> > > More about integration tests, how to build, run, test them and
> integrate
> > > them in our CI can be found here:
> > >
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > > - happy to help if you are stuck with it.
> > >
> > > J.
> > >
> > >
> > > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <
> ahmad.farhan9715@gmail.com
> > >
> > > wrote:
> > >
> > > > I pushed changes to move the provider into the “apache” directory.
> > After
> > > > updating the class references across the project, I re-tested and all
> > tests
> > > > passed.
> > > >
> > > > Regarding the use of Gremlin (or another graph query language like
> > Cypher
> > > > and SPARQL) for a common package approach, here are my thoughts on
> the
> > pros
> > > > and cons:
> > > >
> > > > pros (I can see only one):
> > > >
> > > >    - Gremlin has been widely adopted by different cloud vendors (e.g.
> > Azure
> > > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
> > self-hosted
> > > >    environments.
> > > >
> > > > cons:
> > > >
> > > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their
> own
> > > >    drivers for executing queries.
> > > >    - To achieve a common abstraction, a wrapper around each driver
> > would be
> > > >    required. Each driver has its own connection parameters,
> underlying
> > > >    protocols, and may need method overrides for compatibility with
> > > > different
> > > >    Python versions.
> > > >    - Not all vendors support every query language; for instance,
> > Gremlin
> > > >    for Neo4j has been deprecated in recent releases, while Cosmos DB
> > does
> > > > not
> > > >    support Cypher or SPARQL.
> > > >
> > > > While it would be ideal to have a unified graph query language and
> > driver
> > > > that works seamlessly across different vendors, such a solution does
> > not
> > > > exist at the moment. In my opinion, implementing provider-specific
> > > > solutions for each query language (Gremlin, Cypher, SPARQL) is more
> > > > realistic and practical given the current landscape.
> > > >
> > > > Happy to discuss further or answer any questions!
> > > >
> > > > Farhan
> > > >
> > > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com>
> > > > wrote:
> > > >
> > > > > I have worked with two different graph database vendors—Azure
> Cosmos
> > DB
> > > > > and Neo4j. During our migration to Neo4j, we discovered that using
> > the
> > > > > Gremlin language wasn’t possible; we were forced to rewrite all our
> > > > queries
> > > > > into Cypher, which is the native language for Neo4j and, in my
> > > > experience,
> > > > > much simpler for querying.
> > > > >
> > > > > This situation highlights a key challenge for a common abstraction:
> > the
> > > > > underlying query languages and connection/authentication mechanisms
> > vary
> > > > > significantly. Gremlin is not only different from Cypher in syntax
> > but is
> > > > > also deprecated for Neo4j (see
> > > > > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
> > > > >
> > > > > The question would be how can the common approach accommodate these
> > > > > different query languages?
> > > > >
> > > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com>
> > wrote:
> > > > >
> > > > >> Without deep looking at the code I love the idea - it's very
> > similar to
> > > > >> what we have for common.sql and common.io - and soon
> > common.messaging
> > > > - I
> > > > >> also - long time ago - suggested common.dataframe that someone
> could
> > > > >> submit
> > > > >> using Apache Ibis:
> > > > >> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l
> -
> > > > >> similarly I believe there was an idea about common.llm ...
> > > > >>
> > > > >> I think the "common" pattern is a great one for Airflow, to build
> > on top
> > > > >> of
> > > > >> "other giants" who build those common abstractions that you can
> > easily
> > > > >> switch between different implementations of various data access
> > layers.
> > > > >>
> > > > >> My suggestion and question - would be however (not very strong on
> > it, I
> > > > >> would love to hear what others think, I know it's been somewhat
> > > > >> contentious
> > > > >> when I started the ibis discussion) - would be to make it
> > > > "common.graph",
> > > > >> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis"
> -
> > just
> > > > >> to
> > > > >> stress that those are not implementations of particular service
> but
> > > > >> opinionated choice of particular technology to do "common"
> > operations.
> > > > >> This
> > > > >> is what essentially "common.io" is . - it should be named
> "fsspec"
> > > > >> provider
> > > > >> if we were to name it by the "library" that implemented it.
> > > > >>
> > > > >> J.
> > > > >>
> > > > >>
> > > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > > ahmad.farhan9715@gmail.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi Everyone,
> > > > >> >
> > > > >> > I’ve created a draft PR (
> > https://github.com/apache/airflow/pull/46977
> > > > )
> > > > >> to
> > > > >> > introduce and discuss a new provider for using Gremlin—the graph
> > > > >> traversal
> > > > >> > language of Apache TinkerPop (more details here:
> > > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is
> supported
> > by
> > > > >> > various
> > > > >> > graph database vendors such as Azure Cosmos DB and Amazon
> Neptune.
> > > > >> > Previously, I had to develop a custom hook to query data from
> > Azure
> > > > >> Cosmos
> > > > >> > DB using Apache Gremlin.
> > > > >> >
> > > > >> > I managed to create a provider and run it locally on the main
> > branch.
> > > > >> > However, I ran into the BaseHook issue (
> > > > >> > https://github.com/apache/airflow/issues/45233) on that branch,
> > so I
> > > > >> ended
> > > > >> > up testing it fully on the v2-10-test branch. The PR should be
> > > > complete,
> > > > >> > but I’ve kept it as a draft for now while we discuss the
> provider.
> > > > >> >
> > > > >> > I’m a new contributor, so I’m especially eager to hear your
> > feedback.
> > > > >> > Comments on the PR is very welcome, and please feel free to
> reach
> > out
> > > > >> with
> > > > >> > any questions via email or Slack.
> > > > >> >
> > > > >> > Thanks,
> > > > >> > Ahmad Farhan
> > > > >> >
> > > > >>
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> > For additional commands, e-mail: dev-help@airflow.apache.org
> >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Jarek Potiuk <ja...@potiuk.com>]

> In the interest of ASF trademarks, I would suggest it be called
"apache/tinkerpop" with "Gremlin" naming reserved for operators and the
like, as it is now with GremlinOperator. I think this makes sense because
it is connecting to TinkerPop-enabled systems via Gremlin. I would
similarly suggest that references to "Apache Gremlin" and the like become
"Apache TinkerPop".

That's an interesting one - indeed TinkerPop is the PMC/ Framework -
Gremlin is the language.

I am not sure we are actually using TinkerPop here - because TinkerPop is
the whole framework - Ahmad, can you explain the relation there - are those
other systems simply implement Gremlin as language or do they use TinkerPop
for something / as a backend?

Because that's a bit of a conceptual difference here. For example in the
provider we are importing https://pypi.org/project/gremlinpython not
"tinkerpop" - and it also does not have tinkerpop as dependency.

I wonder if Gremlin is also a Trademark by Apache ? Maybe we should ask
tinkerpop PMC what they think about it?


J.


On Wed, Feb 26, 2025 at 4:55 PM Stephen Mallette <sp...@apache.org>
wrote:

>
>
> On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> > Yeah . `apache/gremlin" seems like a better option then. Does anyone have
> > anything against it?
>
> In the interest of ASF trademarks, I would suggest it be called
> "apache/tinkerpop" with "Gremlin" naming reserved for operators and the
> like, as it is now with GremlinOperator. I think this makes sense because
> it is connecting to TinkerPop-enabled systems via Gremlin. I would
> similarly suggest that references to "Apache Gremlin" and the like become
> "Apache TinkerPop".
>
> > I think we are pretty happy with accepting "other
> > apache" projects as providers, so I see no issue with Gremlin - knowing
> > that we can always reach out to our friendly Apache Community in case of
> > any issues. So - unless we do not hear any "opposition" in a few days, I
> > think it would make sense if you start `[LAZY CONSENSUS]` thread -
> > without a need for `[VOTE]` thread.
> >
> > One thing though that I would love to have - is to also have an
> integration
> > test if possible (we had it with apache.kafka for example) - those are
> > tests that could run **some** graphdb database locally (via
> docker-compose)
> > and run a very rudimentary checks against a "real" database, not a mocked
> > call. That would make it more robust.
> >
> > More about integration tests, how to build, run, test them and integrate
> > them in our CI can be found here:
> >
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> > - happy to help if you are stuck with it.
> >
> > J.
> >
> >
> > On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <ahmad.farhan9715@gmail.com
> >
> > wrote:
> >
> > > I pushed changes to move the provider into the “apache” directory.
> After
> > > updating the class references across the project, I re-tested and all
> tests
> > > passed.
> > >
> > > Regarding the use of Gremlin (or another graph query language like
> Cypher
> > > and SPARQL) for a common package approach, here are my thoughts on the
> pros
> > > and cons:
> > >
> > > pros (I can see only one):
> > >
> > >    - Gremlin has been widely adopted by different cloud vendors (e.g.
> Azure
> > >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
> self-hosted
> > >    environments.
> > >
> > > cons:
> > >
> > >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their own
> > >    drivers for executing queries.
> > >    - To achieve a common abstraction, a wrapper around each driver
> would be
> > >    required. Each driver has its own connection parameters, underlying
> > >    protocols, and may need method overrides for compatibility with
> > > different
> > >    Python versions.
> > >    - Not all vendors support every query language; for instance,
> Gremlin
> > >    for Neo4j has been deprecated in recent releases, while Cosmos DB
> does
> > > not
> > >    support Cypher or SPARQL.
> > >
> > > While it would be ideal to have a unified graph query language and
> driver
> > > that works seamlessly across different vendors, such a solution does
> not
> > > exist at the moment. In my opinion, implementing provider-specific
> > > solutions for each query language (Gremlin, Cypher, SPARQL) is more
> > > realistic and practical given the current landscape.
> > >
> > > Happy to discuss further or answer any questions!
> > >
> > > Farhan
> > >
> > > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> ahmad.farhan9715@gmail.com>
> > > wrote:
> > >
> > > > I have worked with two different graph database vendors—Azure Cosmos
> DB
> > > > and Neo4j. During our migration to Neo4j, we discovered that using
> the
> > > > Gremlin language wasn’t possible; we were forced to rewrite all our
> > > queries
> > > > into Cypher, which is the native language for Neo4j and, in my
> > > experience,
> > > > much simpler for querying.
> > > >
> > > > This situation highlights a key challenge for a common abstraction:
> the
> > > > underlying query languages and connection/authentication mechanisms
> vary
> > > > significantly. Gremlin is not only different from Cypher in syntax
> but is
> > > > also deprecated for Neo4j (see
> > > > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
> > > >
> > > > The question would be how can the common approach accommodate these
> > > > different query languages?
> > > >
> > > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com>
> wrote:
> > > >
> > > >> Without deep looking at the code I love the idea - it's very
> similar to
> > > >> what we have for common.sql and common.io - and soon
> common.messaging
> > > - I
> > > >> also - long time ago - suggested common.dataframe that someone could
> > > >> submit
> > > >> using Apache Ibis:
> > > >> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
> > > >> similarly I believe there was an idea about common.llm ...
> > > >>
> > > >> I think the "common" pattern is a great one for Airflow, to build
> on top
> > > >> of
> > > >> "other giants" who build those common abstractions that you can
> easily
> > > >> switch between different implementations of various data access
> layers.
> > > >>
> > > >> My suggestion and question - would be however (not very strong on
> it, I
> > > >> would love to hear what others think, I know it's been somewhat
> > > >> contentious
> > > >> when I started the ibis discussion) - would be to make it
> > > "common.graph",
> > > >> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" -
> just
> > > >> to
> > > >> stress that those are not implementations of particular service but
> > > >> opinionated choice of particular technology to do "common"
> operations.
> > > >> This
> > > >> is what essentially "common.io" is . - it should be named "fsspec"
> > > >> provider
> > > >> if we were to name it by the "library" that implemented it.
> > > >>
> > > >> J.
> > > >>
> > > >>
> > > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > > ahmad.farhan9715@gmail.com>
> > > >> wrote:
> > > >>
> > > >> > Hi Everyone,
> > > >> >
> > > >> > I’ve created a draft PR (
> https://github.com/apache/airflow/pull/46977
> > > )
> > > >> to
> > > >> > introduce and discuss a new provider for using Gremlin—the graph
> > > >> traversal
> > > >> > language of Apache TinkerPop (more details here:
> > > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported
> by
> > > >> > various
> > > >> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> > > >> > Previously, I had to develop a custom hook to query data from
> Azure
> > > >> Cosmos
> > > >> > DB using Apache Gremlin.
> > > >> >
> > > >> > I managed to create a provider and run it locally on the main
> branch.
> > > >> > However, I ran into the BaseHook issue (
> > > >> > https://github.com/apache/airflow/issues/45233) on that branch,
> so I
> > > >> ended
> > > >> > up testing it fully on the v2-10-test branch. The PR should be
> > > complete,
> > > >> > but I’ve kept it as a draft for now while we discuss the provider.
> > > >> >
> > > >> > I’m a new contributor, so I’m especially eager to hear your
> feedback.
> > > >> > Comments on the PR is very welcome, and please feel free to reach
> out
> > > >> with
> > > >> > any questions via email or Slack.
> > > >> >
> > > >> > Thanks,
> > > >> > Ahmad Farhan
> > > >> >
> > > >>
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
> For additional commands, e-mail: dev-help@airflow.apache.org
>
>

Re: [DISCUSS] New Provider for Gremlin

[Stephen Mallette <sp...@apache.org>]

On 2025/02/26 12:38:02 Jarek Potiuk wrote:
> Yeah . `apache/gremlin" seems like a better option then. Does anyone have
> anything against it? 

In the interest of ASF trademarks, I would suggest it be called "apache/tinkerpop" with "Gremlin" naming reserved for operators and the like, as it is now with GremlinOperator. I think this makes sense because it is connecting to TinkerPop-enabled systems via Gremlin. I would similarly suggest that references to "Apache Gremlin" and the like become "Apache TinkerPop".

> I think we are pretty happy with accepting "other
> apache" projects as providers, so I see no issue with Gremlin - knowing
> that we can always reach out to our friendly Apache Community in case of
> any issues. So - unless we do not hear any "opposition" in a few days, I
> think it would make sense if you start `[LAZY CONSENSUS]` thread -
> without a need for `[VOTE]` thread.
> 
> One thing though that I would love to have - is to also have an integration
> test if possible (we had it with apache.kafka for example) - those are
> tests that could run **some** graphdb database locally (via docker-compose)
> and run a very rudimentary checks against a "real" database, not a mocked
> call. That would make it more robust.
> 
> More about integration tests, how to build, run, test them and integrate
> them in our CI can be found here:
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> - happy to help if you are stuck with it.
> 
> J.
> 
> 
> On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <ah...@gmail.com>
> wrote:
> 
> > I pushed changes to move the provider into the “apache” directory. After
> > updating the class references across the project, I re-tested and all tests
> > passed.
> >
> > Regarding the use of Gremlin (or another graph query language like Cypher
> > and SPARQL) for a common package approach, here are my thoughts on the pros
> > and cons:
> >
> > pros (I can see only one):
> >
> >    - Gremlin has been widely adopted by different cloud vendors (e.g. Azure
> >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in self-hosted
> >    environments.
> >
> > cons:
> >
> >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their own
> >    drivers for executing queries.
> >    - To achieve a common abstraction, a wrapper around each driver would be
> >    required. Each driver has its own connection parameters, underlying
> >    protocols, and may need method overrides for compatibility with
> > different
> >    Python versions.
> >    - Not all vendors support every query language; for instance, Gremlin
> >    for Neo4j has been deprecated in recent releases, while Cosmos DB does
> > not
> >    support Cypher or SPARQL.
> >
> > While it would be ideal to have a unified graph query language and driver
> > that works seamlessly across different vendors, such a solution does not
> > exist at the moment. In my opinion, implementing provider-specific
> > solutions for each query language (Gremlin, Cypher, SPARQL) is more
> > realistic and practical given the current landscape.
> >
> > Happy to discuss further or answer any questions!
> >
> > Farhan
> >
> > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <ah...@gmail.com>
> > wrote:
> >
> > > I have worked with two different graph database vendors—Azure Cosmos DB
> > > and Neo4j. During our migration to Neo4j, we discovered that using the
> > > Gremlin language wasn’t possible; we were forced to rewrite all our
> > queries
> > > into Cypher, which is the native language for Neo4j and, in my
> > experience,
> > > much simpler for querying.
> > >
> > > This situation highlights a key challenge for a common abstraction: the
> > > underlying query languages and connection/authentication mechanisms vary
> > > significantly. Gremlin is not only different from Cypher in syntax but is
> > > also deprecated for Neo4j (see
> > > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
> > >
> > > The question would be how can the common approach accommodate these
> > > different query languages?
> > >
> > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> Without deep looking at the code I love the idea - it's very similar to
> > >> what we have for common.sql and common.io - and soon common.messaging
> > - I
> > >> also - long time ago - suggested common.dataframe that someone could
> > >> submit
> > >> using Apache Ibis:
> > >> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
> > >> similarly I believe there was an idea about common.llm ...
> > >>
> > >> I think the "common" pattern is a great one for Airflow, to build on top
> > >> of
> > >> "other giants" who build those common abstractions that you can easily
> > >> switch between different implementations of various data access layers.
> > >>
> > >> My suggestion and question - would be however (not very strong on it, I
> > >> would love to hear what others think, I know it's been somewhat
> > >> contentious
> > >> when I started the ibis discussion) - would be to make it
> > "common.graph",
> > >> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" - just
> > >> to
> > >> stress that those are not implementations of particular service but
> > >> opinionated choice of particular technology to do "common" operations.
> > >> This
> > >> is what essentially "common.io" is . - it should be named "fsspec"
> > >> provider
> > >> if we were to name it by the "library" that implemented it.
> > >>
> > >> J.
> > >>
> > >>
> > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Everyone,
> > >> >
> > >> > I’ve created a draft PR (https://github.com/apache/airflow/pull/46977
> > )
> > >> to
> > >> > introduce and discuss a new provider for using Gremlin—the graph
> > >> traversal
> > >> > language of Apache TinkerPop (more details here:
> > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
> > >> > various
> > >> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> > >> > Previously, I had to develop a custom hook to query data from Azure
> > >> Cosmos
> > >> > DB using Apache Gremlin.
> > >> >
> > >> > I managed to create a provider and run it locally on the main branch.
> > >> > However, I ran into the BaseHook issue (
> > >> > https://github.com/apache/airflow/issues/45233) on that branch, so I
> > >> ended
> > >> > up testing it fully on the v2-10-test branch. The PR should be
> > complete,
> > >> > but I’ve kept it as a draft for now while we discuss the provider.
> > >> >
> > >> > I’m a new contributor, so I’m especially eager to hear your feedback.
> > >> > Comments on the PR is very welcome, and please feel free to reach out
> > >> with
> > >> > any questions via email or Slack.
> > >> >
> > >> > Thanks,
> > >> > Ahmad Farhan
> > >> >
> > >>
> > >
> >
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@airflow.apache.org
For additional commands, e-mail: dev-help@airflow.apache.org

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

I was thinking integration testing might be something to consider for
testing this provider, I will have a read through the link and implement
that. Thanks Jarek

Farhan

On Wed, Feb 26, 2025 at 12:40 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Yeah . `apache/gremlin" seems like a better option then. Does anyone have
> anything against it? I think we are pretty happy with accepting "other
> apache" projects as providers, so I see no issue with Gremlin - knowing
> that we can always reach out to our friendly Apache Community in case of
> any issues. So - unless we do not hear any "opposition" in a few days, I
> think it would make sense if you start `[LAZY CONSENSUS]` thread -
> without a need for `[VOTE]` thread.
>
> One thing though that I would love to have - is to also have an integration
> test if possible (we had it with apache.kafka for example) - those are
> tests that could run **some** graphdb database locally (via docker-compose)
> and run a very rudimentary checks against a "real" database, not a mocked
> call. That would make it more robust.
>
> More about integration tests, how to build, run, test them and integrate
> them in our CI can be found here:
>
> https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
> - happy to help if you are stuck with it.
>
> J.
>
>
> On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <ah...@gmail.com>
> wrote:
>
> > I pushed changes to move the provider into the “apache” directory. After
> > updating the class references across the project, I re-tested and all
> tests
> > passed.
> >
> > Regarding the use of Gremlin (or another graph query language like Cypher
> > and SPARQL) for a common package approach, here are my thoughts on the
> pros
> > and cons:
> >
> > pros (I can see only one):
> >
> >    - Gremlin has been widely adopted by different cloud vendors (e.g.
> Azure
> >    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in
> self-hosted
> >    environments.
> >
> > cons:
> >
> >    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their own
> >    drivers for executing queries.
> >    - To achieve a common abstraction, a wrapper around each driver would
> be
> >    required. Each driver has its own connection parameters, underlying
> >    protocols, and may need method overrides for compatibility with
> > different
> >    Python versions.
> >    - Not all vendors support every query language; for instance, Gremlin
> >    for Neo4j has been deprecated in recent releases, while Cosmos DB does
> > not
> >    support Cypher or SPARQL.
> >
> > While it would be ideal to have a unified graph query language and driver
> > that works seamlessly across different vendors, such a solution does not
> > exist at the moment. In my opinion, implementing provider-specific
> > solutions for each query language (Gremlin, Cypher, SPARQL) is more
> > realistic and practical given the current landscape.
> >
> > Happy to discuss further or answer any questions!
> >
> > Farhan
> >
> > On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <
> ahmad.farhan9715@gmail.com>
> > wrote:
> >
> > > I have worked with two different graph database vendors—Azure Cosmos DB
> > > and Neo4j. During our migration to Neo4j, we discovered that using the
> > > Gremlin language wasn’t possible; we were forced to rewrite all our
> > queries
> > > into Cypher, which is the native language for Neo4j and, in my
> > experience,
> > > much simpler for querying.
> > >
> > > This situation highlights a key challenge for a common abstraction: the
> > > underlying query languages and connection/authentication mechanisms
> vary
> > > significantly. Gremlin is not only different from Cypher in syntax but
> is
> > > also deprecated for Neo4j (see
> > > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
> > >
> > > The question would be how can the common approach accommodate these
> > > different query languages?
> > >
> > > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> > >
> > >> Without deep looking at the code I love the idea - it's very similar
> to
> > >> what we have for common.sql and common.io - and soon common.messaging
> > - I
> > >> also - long time ago - suggested common.dataframe that someone could
> > >> submit
> > >> using Apache Ibis:
> > >> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
> > >> similarly I believe there was an idea about common.llm ...
> > >>
> > >> I think the "common" pattern is a great one for Airflow, to build on
> top
> > >> of
> > >> "other giants" who build those common abstractions that you can easily
> > >> switch between different implementations of various data access
> layers.
> > >>
> > >> My suggestion and question - would be however (not very strong on it,
> I
> > >> would love to hear what others think, I know it's been somewhat
> > >> contentious
> > >> when I started the ibis discussion) - would be to make it
> > "common.graph",
> > >> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" -
> just
> > >> to
> > >> stress that those are not implementations of particular service but
> > >> opinionated choice of particular technology to do "common" operations.
> > >> This
> > >> is what essentially "common.io" is . - it should be named "fsspec"
> > >> provider
> > >> if we were to name it by the "library" that implemented it.
> > >>
> > >> J.
> > >>
> > >>
> > >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> > ahmad.farhan9715@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Everyone,
> > >> >
> > >> > I’ve created a draft PR (
> https://github.com/apache/airflow/pull/46977
> > )
> > >> to
> > >> > introduce and discuss a new provider for using Gremlin—the graph
> > >> traversal
> > >> > language of Apache TinkerPop (more details here:
> > >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
> > >> > various
> > >> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> > >> > Previously, I had to develop a custom hook to query data from Azure
> > >> Cosmos
> > >> > DB using Apache Gremlin.
> > >> >
> > >> > I managed to create a provider and run it locally on the main
> branch.
> > >> > However, I ran into the BaseHook issue (
> > >> > https://github.com/apache/airflow/issues/45233) on that branch, so
> I
> > >> ended
> > >> > up testing it fully on the v2-10-test branch. The PR should be
> > complete,
> > >> > but I’ve kept it as a draft for now while we discuss the provider.
> > >> >
> > >> > I’m a new contributor, so I’m especially eager to hear your
> feedback.
> > >> > Comments on the PR is very welcome, and please feel free to reach
> out
> > >> with
> > >> > any questions via email or Slack.
> > >> >
> > >> > Thanks,
> > >> > Ahmad Farhan
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Jarek Potiuk <ja...@potiuk.com>]

Yeah . `apache/gremlin" seems like a better option then. Does anyone have
anything against it? I think we are pretty happy with accepting "other
apache" projects as providers, so I see no issue with Gremlin - knowing
that we can always reach out to our friendly Apache Community in case of
any issues. So - unless we do not hear any "opposition" in a few days, I
think it would make sense if you start `[LAZY CONSENSUS]` thread -
without a need for `[VOTE]` thread.

One thing though that I would love to have - is to also have an integration
test if possible (we had it with apache.kafka for example) - those are
tests that could run **some** graphdb database locally (via docker-compose)
and run a very rudimentary checks against a "real" database, not a mocked
call. That would make it more robust.

More about integration tests, how to build, run, test them and integrate
them in our CI can be found here:
https://github.com/apache/airflow/blob/main/contributing-docs/testing/integration_tests.rst
- happy to help if you are stuck with it.

J.


On Wed, Feb 26, 2025 at 1:25 PM Ahmad Farhan <ah...@gmail.com>
wrote:

> I pushed changes to move the provider into the “apache” directory. After
> updating the class references across the project, I re-tested and all tests
> passed.
>
> Regarding the use of Gremlin (or another graph query language like Cypher
> and SPARQL) for a common package approach, here are my thoughts on the pros
> and cons:
>
> pros (I can see only one):
>
>    - Gremlin has been widely adopted by different cloud vendors (e.g. Azure
>    Cosmos DB with Apache Gremlin and AWS Neptune) as well as in self-hosted
>    environments.
>
> cons:
>
>    - Gremlin, Cypher (native for Neo4j) and SPARQL each have their own
>    drivers for executing queries.
>    - To achieve a common abstraction, a wrapper around each driver would be
>    required. Each driver has its own connection parameters, underlying
>    protocols, and may need method overrides for compatibility with
> different
>    Python versions.
>    - Not all vendors support every query language; for instance, Gremlin
>    for Neo4j has been deprecated in recent releases, while Cosmos DB does
> not
>    support Cypher or SPARQL.
>
> While it would be ideal to have a unified graph query language and driver
> that works seamlessly across different vendors, such a solution does not
> exist at the moment. In my opinion, implementing provider-specific
> solutions for each query language (Gremlin, Cypher, SPARQL) is more
> realistic and practical given the current landscape.
>
> Happy to discuss further or answer any questions!
>
> Farhan
>
> On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <ah...@gmail.com>
> wrote:
>
> > I have worked with two different graph database vendors—Azure Cosmos DB
> > and Neo4j. During our migration to Neo4j, we discovered that using the
> > Gremlin language wasn’t possible; we were forced to rewrite all our
> queries
> > into Cypher, which is the native language for Neo4j and, in my
> experience,
> > much simpler for querying.
> >
> > This situation highlights a key challenge for a common abstraction: the
> > underlying query languages and connection/authentication mechanisms vary
> > significantly. Gremlin is not only different from Cypher in syntax but is
> > also deprecated for Neo4j (see
> > https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
> >
> > The question would be how can the common approach accommodate these
> > different query languages?
> >
> > On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
> >
> >> Without deep looking at the code I love the idea - it's very similar to
> >> what we have for common.sql and common.io - and soon common.messaging
> - I
> >> also - long time ago - suggested common.dataframe that someone could
> >> submit
> >> using Apache Ibis:
> >> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
> >> similarly I believe there was an idea about common.llm ...
> >>
> >> I think the "common" pattern is a great one for Airflow, to build on top
> >> of
> >> "other giants" who build those common abstractions that you can easily
> >> switch between different implementations of various data access layers.
> >>
> >> My suggestion and question - would be however (not very strong on it, I
> >> would love to hear what others think, I know it's been somewhat
> >> contentious
> >> when I started the ibis discussion) - would be to make it
> "common.graph",
> >> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" - just
> >> to
> >> stress that those are not implementations of particular service but
> >> opinionated choice of particular technology to do "common" operations.
> >> This
> >> is what essentially "common.io" is . - it should be named "fsspec"
> >> provider
> >> if we were to name it by the "library" that implemented it.
> >>
> >> J.
> >>
> >>
> >> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <
> ahmad.farhan9715@gmail.com>
> >> wrote:
> >>
> >> > Hi Everyone,
> >> >
> >> > I’ve created a draft PR (https://github.com/apache/airflow/pull/46977
> )
> >> to
> >> > introduce and discuss a new provider for using Gremlin—the graph
> >> traversal
> >> > language of Apache TinkerPop (more details here:
> >> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
> >> > various
> >> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> >> > Previously, I had to develop a custom hook to query data from Azure
> >> Cosmos
> >> > DB using Apache Gremlin.
> >> >
> >> > I managed to create a provider and run it locally on the main branch.
> >> > However, I ran into the BaseHook issue (
> >> > https://github.com/apache/airflow/issues/45233) on that branch, so I
> >> ended
> >> > up testing it fully on the v2-10-test branch. The PR should be
> complete,
> >> > but I’ve kept it as a draft for now while we discuss the provider.
> >> >
> >> > I’m a new contributor, so I’m especially eager to hear your feedback.
> >> > Comments on the PR is very welcome, and please feel free to reach out
> >> with
> >> > any questions via email or Slack.
> >> >
> >> > Thanks,
> >> > Ahmad Farhan
> >> >
> >>
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

I pushed changes to move the provider into the “apache” directory. After
updating the class references across the project, I re-tested and all tests
passed.

Regarding the use of Gremlin (or another graph query language like Cypher
and SPARQL) for a common package approach, here are my thoughts on the pros
and cons:

pros (I can see only one):

   - Gremlin has been widely adopted by different cloud vendors (e.g. Azure
   Cosmos DB with Apache Gremlin and AWS Neptune) as well as in self-hosted
   environments.

cons:

   - Gremlin, Cypher (native for Neo4j) and SPARQL each have their own
   drivers for executing queries.
   - To achieve a common abstraction, a wrapper around each driver would be
   required. Each driver has its own connection parameters, underlying
   protocols, and may need method overrides for compatibility with different
   Python versions.
   - Not all vendors support every query language; for instance, Gremlin
   for Neo4j has been deprecated in recent releases, while Cosmos DB does not
   support Cypher or SPARQL.

While it would be ideal to have a unified graph query language and driver
that works seamlessly across different vendors, such a solution does not
exist at the moment. In my opinion, implementing provider-specific
solutions for each query language (Gremlin, Cypher, SPARQL) is more
realistic and practical given the current landscape.

Happy to discuss further or answer any questions!

Farhan

On Mon, Feb 24, 2025 at 11:33 AM Ahmad Farhan <ah...@gmail.com>
wrote:

> I have worked with two different graph database vendors—Azure Cosmos DB
> and Neo4j. During our migration to Neo4j, we discovered that using the
> Gremlin language wasn’t possible; we were forced to rewrite all our queries
> into Cypher, which is the native language for Neo4j and, in my experience,
> much simpler for querying.
>
> This situation highlights a key challenge for a common abstraction: the
> underlying query languages and connection/authentication mechanisms vary
> significantly. Gremlin is not only different from Cypher in syntax but is
> also deprecated for Neo4j (see
> https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).
>
> The question would be how can the common approach accommodate these
> different query languages?
>
> On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:
>
>> Without deep looking at the code I love the idea - it's very similar to
>> what we have for common.sql and common.io - and soon common.messaging - I
>> also - long time ago - suggested common.dataframe that someone could
>> submit
>> using Apache Ibis:
>> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
>> similarly I believe there was an idea about common.llm ...
>>
>> I think the "common" pattern is a great one for Airflow, to build on top
>> of
>> "other giants" who build those common abstractions that you can easily
>> switch between different implementations of various data access layers.
>>
>> My suggestion and question - would be however (not very strong on it, I
>> would love to hear what others think, I know it's been somewhat
>> contentious
>> when I started the ibis discussion) - would be to make it "common.graph",
>> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" - just
>> to
>> stress that those are not implementations of particular service but
>> opinionated choice of particular technology to do "common" operations.
>> This
>> is what essentially "common.io" is . - it should be named "fsspec"
>> provider
>> if we were to name it by the "library" that implemented it.
>>
>> J.
>>
>>
>> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <ah...@gmail.com>
>> wrote:
>>
>> > Hi Everyone,
>> >
>> > I’ve created a draft PR (https://github.com/apache/airflow/pull/46977)
>> to
>> > introduce and discuss a new provider for using Gremlin—the graph
>> traversal
>> > language of Apache TinkerPop (more details here:
>> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
>> > various
>> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
>> > Previously, I had to develop a custom hook to query data from Azure
>> Cosmos
>> > DB using Apache Gremlin.
>> >
>> > I managed to create a provider and run it locally on the main branch.
>> > However, I ran into the BaseHook issue (
>> > https://github.com/apache/airflow/issues/45233) on that branch, so I
>> ended
>> > up testing it fully on the v2-10-test branch. The PR should be complete,
>> > but I’ve kept it as a draft for now while we discuss the provider.
>> >
>> > I’m a new contributor, so I’m especially eager to hear your feedback.
>> > Comments on the PR is very welcome, and please feel free to reach out
>> with
>> > any questions via email or Slack.
>> >
>> > Thanks,
>> > Ahmad Farhan
>> >
>>
>

Re: [DISCUSS] New Provider for Gremlin

[Ahmad Farhan <ah...@gmail.com>]

I have worked with two different graph database vendors—Azure Cosmos DB and
Neo4j. During our migration to Neo4j, we discovered that using the Gremlin
language wasn’t possible; we were forced to rewrite all our queries into
Cypher, which is the native language for Neo4j and, in my experience, much
simpler for querying.

This situation highlights a key challenge for a common abstraction: the
underlying query languages and connection/authentication mechanisms vary
significantly. Gremlin is not only different from Cypher in syntax but is
also deprecated for Neo4j (see
https://tinkerpop.apache.org/docs/3.7.3/reference/#neo4j-gremlin).

The question would be how can the common approach accommodate these
different query languages?

On Fri, Feb 21, 2025 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote:

> Without deep looking at the code I love the idea - it's very similar to
> what we have for common.sql and common.io - and soon common.messaging - I
> also - long time ago - suggested common.dataframe that someone could submit
> using Apache Ibis:
> https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
> similarly I believe there was an idea about common.llm ...
>
> I think the "common" pattern is a great one for Airflow, to build on top of
> "other giants" who build those common abstractions that you can easily
> switch between different implementations of various data access layers.
>
> My suggestion and question - would be however (not very strong on it, I
> would love to hear what others think, I know it's been somewhat contentious
> when I started the ibis discussion) - would be to make it "common.graph",
> "common.dataframe" - instead of "apache.gremlin" or "apache.ibis" - just to
> stress that those are not implementations of particular service but
> opinionated choice of particular technology to do "common" operations. This
> is what essentially "common.io" is . - it should be named "fsspec"
> provider
> if we were to name it by the "library" that implemented it.
>
> J.
>
>
> On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <ah...@gmail.com>
> wrote:
>
> > Hi Everyone,
> >
> > I’ve created a draft PR (https://github.com/apache/airflow/pull/46977)
> to
> > introduce and discuss a new provider for using Gremlin—the graph
> traversal
> > language of Apache TinkerPop (more details here:
> > https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
> > various
> > graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> > Previously, I had to develop a custom hook to query data from Azure
> Cosmos
> > DB using Apache Gremlin.
> >
> > I managed to create a provider and run it locally on the main branch.
> > However, I ran into the BaseHook issue (
> > https://github.com/apache/airflow/issues/45233) on that branch, so I
> ended
> > up testing it fully on the v2-10-test branch. The PR should be complete,
> > but I’ve kept it as a draft for now while we discuss the provider.
> >
> > I’m a new contributor, so I’m especially eager to hear your feedback.
> > Comments on the PR is very welcome, and please feel free to reach out
> with
> > any questions via email or Slack.
> >
> > Thanks,
> > Ahmad Farhan
> >
>

Re: [DISCUSS] New Provider for Gremlin

[Jarek Potiuk <ja...@potiuk.com>]

Without deep looking at the code I love the idea - it's very similar to
what we have for common.sql and common.io - and soon common.messaging - I
also - long time ago - suggested common.dataframe that someone could submit
using Apache Ibis:
https://lists.apache.org/thread/qx3yh6h0l6jb0kh3fz9q95b3x5b4001l  -
similarly I believe there was an idea about common.llm ...

I think the "common" pattern is a great one for Airflow, to build on top of
"other giants" who build those common abstractions that you can easily
switch between different implementations of various data access layers.

My suggestion and question - would be however (not very strong on it, I
would love to hear what others think, I know it's been somewhat contentious
when I started the ibis discussion) - would be to make it "common.graph",
"common.dataframe" - instead of "apache.gremlin" or "apache.ibis" - just to
stress that those are not implementations of particular service but
opinionated choice of particular technology to do "common" operations. This
is what essentially "common.io" is . - it should be named "fsspec" provider
if we were to name it by the "library" that implemented it.

J.


On Fri, Feb 21, 2025 at 8:22 PM Ahmad Farhan <ah...@gmail.com>
wrote:

> Hi Everyone,
>
> I’ve created a draft PR (https://github.com/apache/airflow/pull/46977) to
> introduce and discuss a new provider for using Gremlin—the graph traversal
> language of Apache TinkerPop (more details here:
> https://tinkerpop.apache.org/gremlin.html). Gremlin is supported by
> various
> graph database vendors such as Azure Cosmos DB and Amazon Neptune.
> Previously, I had to develop a custom hook to query data from Azure Cosmos
> DB using Apache Gremlin.
>
> I managed to create a provider and run it locally on the main branch.
> However, I ran into the BaseHook issue (
> https://github.com/apache/airflow/issues/45233) on that branch, so I ended
> up testing it fully on the v2-10-test branch. The PR should be complete,
> but I’ve kept it as a draft for now while we discuss the provider.
>
> I’m a new contributor, so I’m especially eager to hear your feedback.
> Comments on the PR is very welcome, and please feel free to reach out with
> any questions via email or Slack.
>
> Thanks,
> Ahmad Farhan
>