You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Aaron199 (via GitHub)" <gi...@apache.org> on 2026/03/30 08:46:47 UTC
[I] [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type [doris]
Aaron199 opened a new issue, #61883:
URL: https://github.com/apache/doris/issues/61883
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
**Environment**
- **Doris version:** 4.0.3
- **Deployment:** Kubernetes (k8s) Operator
- **Cluster Type:** `DorisDisaggregatedCluster` (Cloud Mode / Storage-Compute separation)
- **Configuration:** Catalog property `"use_arrow_flight" = "true"` is set.
### What's Wrong?
**Describe the bug**
In Doris 4.0.3, when using a cross-cluster Catalog (`type = 'doris'`) with the Arrow Flight protocol enabled (`use_arrow_flight = 'true'`) to read data from a remote Doris cluster, querying a table that contains a `Variant` column fails. The query throws a `NOT_IMPLEMENTED_ERROR`. It appears the underlying Arrow reader currently does not support the serialization/deserialization of the `Variant` data type.
**To Reproduce**
1. Create a Catalog connecting to a remote Doris cluster and force enable Arrow Flight:
```sql
CREATE CATALOG remote_doris PROPERTIES (
"type" = "doris",
"use_arrow_flight" = "true",
"fe_http_hosts" = "...",
"fe_arrow_hosts" = "...",
"fe_thrift_hosts" = "...",
"user" = "...",
"password" = "..."
);
```
2. Ensure the target table being queried on the remote cluster contains a column of type `Variant` (e.g., `ext_fields`).
3. Execute a query from the local cluster via this Catalog:
```sql
SELECT ext_fields FROM remote_doris.db.table LIMIT 10;
```
4. The following error is triggered:
```text
ERROR 1105 (HY000): errCode = 2, detailMessage = (...)[NOT_IMPLEMENTED_ERROR]read_column_from_arrow with type variant. cur path: /dummyPath
```
*(Note: Even if attempting to use `CAST(ext_fields AS STRING)` in the query, the conversion cannot be pushed down before the error occurs, triggering the same interception error.)*
### What You Expected?
The Arrow Flight transmission link should natively support the cross-cluster reading and parsing of `Variant` types, allowing the query to return data successfully without throwing a `NOT_IMPLEMENTED_ERROR`.
### How to Reproduce?
_No response_
### Anything Else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type [doris]
Posted by "Arun12-sketch (via GitHub)" <gi...@apache.org>.
Arun12-sketch commented on issue #61883:
URL: https://github.com/apache/doris/issues/61883#issuecomment-4159638413
hey @Aaron199 , I looked into this and the issue is that Variant type has no Arrow serialization support yet. Since Variant is stored as JSON internally, I think we can serialize it as Arrow LargeUtf8 string during flight transmission. Would need to update arrow_column_to_doris_column.cpp and doris_column_to_arrow_column.cpp to handle TYPE_VARIANT. Does this approach sound good before I start working on a PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type [doris]
Posted by "Aaron199 (via GitHub)" <gi...@apache.org>.
Aaron199 commented on issue #61883:
URL: https://github.com/apache/doris/issues/61883#issuecomment-4349488785
Hi @Arun12-sketch ,
Thanks for the proposal! Using LargeUtf8 for Variant serialization is a solid starting point for functional support. However, from a performance perspective, I have a few thoughts:
1. **CPU Overhead:** Since Variant is JSON internally, constant string conversion and JSON parsing might become a CPU bottleneck during high-throughput cross-cluster reads.
2. **Data Volume:** JSON strings typically consume more bandwidth compared to optimized binary formats.
**My suggestion:**
* This approach is great as a first step to fix the NOT_IMPLEMENTED_ERROR.
* In the long run, should we consider using Arrow Extension Types or a more efficient binary mapping to minimize serialization overhead?
If the goal is to provide a quick fix now, I'm all for it. We can treat this as V1 and look into performance optimizations later. What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type [doris]
Posted by "yuvarajl (via GitHub)" <gi...@apache.org>.
yuvarajl commented on issue #61883:
URL: https://github.com/apache/doris/issues/61883#issuecomment-4259448254
We are hit by this bug.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] Cross-cluster Catalog with Arrow Flight fails to read Variant type [doris]
Posted by "Arun12-sketch (via GitHub)" <gi...@apache.org>.
Arun12-sketch commented on issue #61883:
URL: https://github.com/apache/doris/issues/61883#issuecomment-4994474487
Hi @Aaron199,
Thanks for the feedback! I completely agree with your assessment regarding the CPU overhead and bandwidth limitations of JSON string conversions.
Treating this as V1 to quickly unblock the NOT_IMPLEMENTED_ERROR makes perfect sense. I'll proceed with implementing the serialization/deserialization using LargeUtf8 to get functional support in place.
Once this is merged, we can definitely track V2 as a separate issue to explore Arrow Extension Types or direct binary JSONB mapping for better performance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org