You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tinkerpop.apache.org by Valentyn Kahamlyk <va...@gmail.com> on 2025/07/22 20:58:21 UTC
[DISCUSS] Proposal for provider-focused interfaces for efficient data retrieval
Hi all,
I would like to discuss this idea for TinkerPop 4.
# Proposal: Provider-focused interfaces for efficient data retrieval
When building Gremlin providers - whether connecting to remote databases,
in‐memory stores, or custom backends - there’s a recurring challenge:
TinkerPop’s core traversal engine asks for full element payloads, while
many providers only need subsets of data (selected properties or specific
edge directions) to satisfy a step. This mismatch leads to:
- Unnecessary data transfer or object materialization
- Wasted CPU cycles filtering out unused fields
- Hard-to-maintain provider code packed with ad hoc “if this step needs X,
fetch Y” logic
Introducing three lightweight interfaces - `PropertiesProcessor`,
`AdjacentProcessor`, and `ElementReader` - offers a clean extension point
that lives in provider code without bloating the TinkerPop core.
---
## Proposed Interfaces
```java
public interface PropertiesProcessor {
enum PropertiesRequirement {
NONE,
CUSTOM,
ALL,
}
// second value is properties
Pair<PropertiesRequirement,String> getPropertiesRequirement();
}
public interface AdjacentProcessor {
enum AdjacentRequirement {
NONE,
IN,
OUT,
BOTH
}
// second value is edge labels. Null if all labels are required.
Pair<AdjacentRequirement, String> getAdjacentRequirement();
}
public interface ElementReader {
void setElementRequirement(Pair<PropertiesRequirement, String>
propertiesRequirement,
Pair<AdjacentRequirement, String>
adjacentRequirement);
}
```
---
## Why This Helps Providers
- **Performance Gains**
By knowing exactly which properties and adjacent edges are required,
providers can:
- Send targeted queries to the backend (e.g., read only needed
columns/properties)
- Avoid loading full element metadata into memory
- Reduce network bandwidth for remote stores
- **Clear Separation of Concerns**
Core TinkerPop remains focused on traversal semantics. Providers
implement these interfaces to translate greedy element requests into
optimized backend calls.
- **Consistent Extension Point**
Instead of scattering `if (step instanceof X)` checks across provider
code, every step that implements `PropertiesProcessor` or
`AdjacentProcessor` exposes its requirements via the same API.
- **Easier Maintenance & Evolution**
Future provider enhancements - like supporting richer filters or
push-down queries - build upon these interfaces. There’s no need to touch
core TinkerPop logic, only provider‐side readers.
---
## Impact on TinkerPop Core
- No new dependencies in the core module
- Default behavior: if a step doesn’t implement these interfaces, providers
fall back to fetching full elements
---
## Conclusion
Adding `PropertiesProcessor`, `AdjacentProcessor`, and `ElementReader` as
provider‐focused interfaces creates a lightweight, uniform way for Gremlin
providers to express data needs. This yields measurable performance
improvements, cleaner code, and a more maintainable separation between
TinkerPop’s traversal engine and its many backend implementations.
Thank you,
Valentyn
Re: [DISCUSS] Proposal for provider-focused interfaces for efficient data retrieval
Posted by Valentyn Kahamlyk <va...@apache.org>.
Hi Andrea,
Example of using these interfaces.
GraphStep implements ElementReader.
This step is almost always overridden by providers, even in TinkerPop.
By implementing ElementReader interface provider can easily understand what needs to be read from storage database.
VertexStep implements ElementReader, AdjacentProcessor, PropertiesProcessor
implementation of AdjacentProcessor can be like
```
public Pair<AdjacentRequirement, String[]> getAdjacentRequirement() {
return Pair.with(
this.direction == Direction.IN ? AdjacentRequirement.IN :
this.direction == Direction.OUT ? AdjacentRequirement.OUT : AdjacentRequirement.BOTH,
this.edgeLabels);
}
@Override
public Pair<PropertiesRequirement, String[]> getPropertiesRequirement() {
return Pair.with(PropertiesRequirement.NONE, null);
}
```
PropertiesStep implements PropertiesProcessor, AdjacentProcessor
this step can uses only a subset of the properties and does not use information about adjacent edges, so implementation can be like
```
@Override
public Pair<AdjacentRequirement, String[]> getAdjacentRequirement() {
return Pair.with(AdjacentRequirement.NONE, null);
}
@Override
public Pair<PropertiesRequirement, String[]> getPropertiesRequirement() {
if (propertyKeys == null) {
return Pair.with(PropertiesRequirement.ALL, null);
} else if (propertyKeys.length == 0) {
return Pair.with(PropertiesRequirement.NONE, null);
} else {
// if there are some keys, then it is CUSTOM
return Pair.with(PropertiesRequirement.CUSTOM, propertyKeys);
}
}
```
Strategy is also needed to spread information about the requirements of the different steps, from back to start.
There are some peculiarities in working with labels (as('a')...select('a')) and child traversers.
Example of optimization for traversal g.V().out().values("name")
PropertiesStep will report its require only "name" property and no edge ids.
VertexStep will use that information to read vertices from db with only required fields/columns/properties.
GraphStep will know to read adjusted edges for direction OUT and skip properties.
Thank you,
Valentyn
On 2025/07/22 22:39:52 Andrea Child wrote:
> Hi Valentyn,
>
> I can see how your proposal would be beneficial for providers. Could you provide an example of step implementation of those interfaces? The example(s) would be helpful to communicate how you envision the proposed interfaces to be used by providers.
>
> Thanks,
>
> Andrea
>
> From: Valentyn Kahamlyk <va...@gmail.com>
> Date: Tuesday, July 22, 2025 at 1:58 PM
> To: dev@tinkerpop.apache.org <de...@tinkerpop.apache.org>
> Subject: [DISCUSS] Proposal for provider-focused interfaces for efficient data retrieval
> Hi all,
>
> I would like to discuss this idea for TinkerPop 4.
>
> # Proposal: Provider-focused interfaces for efficient data retrieval
>
> When building Gremlin providers - whether connecting to remote databases,
> in‐memory stores, or custom backends - there’s a recurring challenge:
> TinkerPop’s core traversal engine asks for full element payloads, while
> many providers only need subsets of data (selected properties or specific
> edge directions) to satisfy a step. This mismatch leads to:
>
> - Unnecessary data transfer or object materialization
> - Wasted CPU cycles filtering out unused fields
> - Hard-to-maintain provider code packed with ad hoc “if this step needs X,
> fetch Y” logic
>
> Introducing three lightweight interfaces - `PropertiesProcessor`,
> `AdjacentProcessor`, and `ElementReader` - offers a clean extension point
> that lives in provider code without bloating the TinkerPop core.
>
> ---
>
> ## Proposed Interfaces
>
> ```java
> public interface PropertiesProcessor {
> enum PropertiesRequirement {
> NONE,
> CUSTOM,
> ALL,
> }
>
> // second value is properties
> Pair<PropertiesRequirement,String> getPropertiesRequirement();
> }
>
> public interface AdjacentProcessor {
> enum AdjacentRequirement {
> NONE,
> IN,
> OUT,
> BOTH
> }
>
> // second value is edge labels. Null if all labels are required.
> Pair<AdjacentRequirement, String> getAdjacentRequirement();
> }
>
> public interface ElementReader {
> void setElementRequirement(Pair<PropertiesRequirement, String>
> propertiesRequirement,
> Pair<AdjacentRequirement, String>
> adjacentRequirement);
> }
> ```
>
> ---
>
> ## Why This Helps Providers
>
> - **Performance Gains**
> By knowing exactly which properties and adjacent edges are required,
> providers can:
> - Send targeted queries to the backend (e.g., read only needed
> columns/properties)
> - Avoid loading full element metadata into memory
> - Reduce network bandwidth for remote stores
>
> - **Clear Separation of Concerns**
> Core TinkerPop remains focused on traversal semantics. Providers
> implement these interfaces to translate greedy element requests into
> optimized backend calls.
>
> - **Consistent Extension Point**
> Instead of scattering `if (step instanceof X)` checks across provider
> code, every step that implements `PropertiesProcessor` or
> `AdjacentProcessor` exposes its requirements via the same API.
>
> - **Easier Maintenance & Evolution**
> Future provider enhancements - like supporting richer filters or
> push-down queries - build upon these interfaces. There’s no need to touch
> core TinkerPop logic, only provider‐side readers.
>
> ---
>
> ## Impact on TinkerPop Core
>
> - No new dependencies in the core module
> - Default behavior: if a step doesn’t implement these interfaces, providers
> fall back to fetching full elements
>
> ---
>
> ## Conclusion
>
> Adding `PropertiesProcessor`, `AdjacentProcessor`, and `ElementReader` as
> provider‐focused interfaces creates a lightweight, uniform way for Gremlin
> providers to express data needs. This yields measurable performance
> improvements, cleaner code, and a more maintainable separation between
> TinkerPop’s traversal engine and its many backend implementations.
>
> Thank you,
> Valentyn
> Warning: The sender of this message could not be validated and may not be the actual sender.
>
Re: [DISCUSS] Proposal for provider-focused interfaces for efficient data retrieval
Posted by Andrea Child <An...@improving.com.INVALID>.
Hi Valentyn,
I can see how your proposal would be beneficial for providers. Could you provide an example of step implementation of those interfaces? The example(s) would be helpful to communicate how you envision the proposed interfaces to be used by providers.
Thanks,
Andrea
From: Valentyn Kahamlyk <va...@gmail.com>
Date: Tuesday, July 22, 2025 at 1:58 PM
To: dev@tinkerpop.apache.org <de...@tinkerpop.apache.org>
Subject: [DISCUSS] Proposal for provider-focused interfaces for efficient data retrieval
Hi all,
I would like to discuss this idea for TinkerPop 4.
# Proposal: Provider-focused interfaces for efficient data retrieval
When building Gremlin providers - whether connecting to remote databases,
in‐memory stores, or custom backends - there’s a recurring challenge:
TinkerPop’s core traversal engine asks for full element payloads, while
many providers only need subsets of data (selected properties or specific
edge directions) to satisfy a step. This mismatch leads to:
- Unnecessary data transfer or object materialization
- Wasted CPU cycles filtering out unused fields
- Hard-to-maintain provider code packed with ad hoc “if this step needs X,
fetch Y” logic
Introducing three lightweight interfaces - `PropertiesProcessor`,
`AdjacentProcessor`, and `ElementReader` - offers a clean extension point
that lives in provider code without bloating the TinkerPop core.
---
## Proposed Interfaces
```java
public interface PropertiesProcessor {
enum PropertiesRequirement {
NONE,
CUSTOM,
ALL,
}
// second value is properties
Pair<PropertiesRequirement,String> getPropertiesRequirement();
}
public interface AdjacentProcessor {
enum AdjacentRequirement {
NONE,
IN,
OUT,
BOTH
}
// second value is edge labels. Null if all labels are required.
Pair<AdjacentRequirement, String> getAdjacentRequirement();
}
public interface ElementReader {
void setElementRequirement(Pair<PropertiesRequirement, String>
propertiesRequirement,
Pair<AdjacentRequirement, String>
adjacentRequirement);
}
```
---
## Why This Helps Providers
- **Performance Gains**
By knowing exactly which properties and adjacent edges are required,
providers can:
- Send targeted queries to the backend (e.g., read only needed
columns/properties)
- Avoid loading full element metadata into memory
- Reduce network bandwidth for remote stores
- **Clear Separation of Concerns**
Core TinkerPop remains focused on traversal semantics. Providers
implement these interfaces to translate greedy element requests into
optimized backend calls.
- **Consistent Extension Point**
Instead of scattering `if (step instanceof X)` checks across provider
code, every step that implements `PropertiesProcessor` or
`AdjacentProcessor` exposes its requirements via the same API.
- **Easier Maintenance & Evolution**
Future provider enhancements - like supporting richer filters or
push-down queries - build upon these interfaces. There’s no need to touch
core TinkerPop logic, only provider‐side readers.
---
## Impact on TinkerPop Core
- No new dependencies in the core module
- Default behavior: if a step doesn’t implement these interfaces, providers
fall back to fetching full elements
---
## Conclusion
Adding `PropertiesProcessor`, `AdjacentProcessor`, and `ElementReader` as
provider‐focused interfaces creates a lightweight, uniform way for Gremlin
providers to express data needs. This yields measurable performance
improvements, cleaner code, and a more maintainable separation between
TinkerPop’s traversal engine and its many backend implementations.
Thank you,
Valentyn
Warning: The sender of this message could not be validated and may not be the actual sender.