You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@camel.apache.org by GitBox <gi...@apache.org> on 2022/11/08 21:13:38 UTC
[GitHub] [camel] christophd opened a new pull request, #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
christophd opened a new pull request, #8694:
URL: https://github.com/apache/camel/pull/8694
WIP do not merge!
This PR is supposed to be a work in progress seeking for guidance and help on implementing multiple data types on components feature as described in https://issues.apache.org/jira/browse/CAMEL-18698
The changes made in this PR should be seen as a POC starting some discussions and refinements of the feature.
The idea is to let the user choose a specific data type that the component is producing as an output. The component itself may offer specific data transformation logic that gets automatically triggered when this data type is explicitly used on an endpoint URI:
```
from("kafka:topic?format=avro").to("aws2-s3:bucketName?format=string")
```
The `format` option tells the component to apply the specific data type as an input or output. The data type may be directly supported by the component or it may be a generic data type that makes use of the Camel TypeConverter logic to transform from one data to another.
In the PR the feature is implemented as a POC for aws2-ddb and aws2-s3 components. The aws2 components often use Java POJO domain objects as input and output so the user needs to know the domain model when interacting with the components.
The idea is to also add support for more generic data types on these components. As an example the user may also provide a generic Json structure as an input instead of the AWS domain object. The component takes care on transforming the data type to the required Java POJO used in the AWS client libraries.
On the output side the aws2-s3 component by default produces byte[] or InputStream output. With the addition of data types the user is able to request another data type such as String. The component makes sure to apply transformation logic from the AWS domain model InputStream implementation to a String.
This feature is beneficial for Camel users that do not have to know about domain specific Java types anymore. Also declarative Camel DSL use cases e.g. used in Camel K and Kamelets benefit from the auto data type conversion as discussed in https://github.com/apache/camel-k/issues/1980
A Kamelet may use the provided data types information and expose this as part of the Kamelet specification.
```
apiVersion: camel.apache.org/v1alpha1
kind: Kamelet
metadata:
name: aws-s3-source
spec:
definition:
title: "AWS S3 Source"
properties:
...
types:
out:
default: binary
binary:
mediaType: application/octet-stream
stream:
mediaType: application/octet-stream
json:
mediaType: application/json
schema:
type: object
required:
- key
- fileContent
properties:
key:
title: Key
description: The S3 file name key
type: string
fileContent:
title: Content
description: The S3 file content as String
type: string
dependencies:
- "camel:jackson"
string:
mediaType: plain/text
avro:
mediaType: application/avro
```
Also the user is able to choose the input/output data type in a binding:
```
apiVersion: camel.apache.org/v1alpha1
kind: KameletBinding
metadata:
name: aws-s3-uri-binding
spec:
source:
ref:
kind: Kamelet
apiVersion: camel.apache.org/v1alpha1
name: aws-s3-source
output:
type: avro
schema:
uri: http://schema-registry/person
properties:
outputFormat: avro
bucketNameOrArn: ${aws.s3.bucketNameOrArn}
sink:
uri: log:info
```
The new data types implementation provided in this PR use a specific SPI annotation `@DataType`. A resolver mechanism `DefaultDataTypeResolver` is capable of doing a lookup of the respective data type implementation with given component scheme and a format name that identifies a data type.
The PR definitely needs more polishing and has some ToDos in the code that seek for guidance and help how to do it the Camel way (e.g. how to do an automatic resource lookup for data type implementations provided by components).
Also it would be nice to have some guidance on the already existing `org.apache.camel.spi.DataType` and how that could be related to this implementation.
Many thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] davsclaus closed pull request #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus closed pull request #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
URL: https://github.com/apache/camel/pull/8694
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #8694:
URL: https://github.com/apache/camel/pull/8694#issuecomment-1307833197
:star2: Thank you for your contribution to the Apache Camel project! :star2:
:warning: Please note that the changes on this PR may be **tested automatically**.
If necessary Apache Camel Committers may access logs and test results in the job summaries!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] github-actions[bot] commented on pull request #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #8694:
URL: https://github.com/apache/camel/pull/8694#issuecomment-1309267772
### Components tested:
| Total | Tested | Failed :x: | Passed :white_check_mark: |
| --- | --- | --- | --- |
| 7 | 7 | 2 | 5 |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [camel] davsclaus commented on pull request #8694: WIP [CAMEL-18698] Add support for multiple input/output data types on components
Posted by "davsclaus (via GitHub)" <gi...@apache.org>.
davsclaus commented on PR #8694:
URL: https://github.com/apache/camel/pull/8694#issuecomment-1608827789
closing this as there is another newer PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@camel.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org