You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Ewen Cheslack-Postava (JIRA)" <ji...@apache.org> on 2017/06/22 16:49:00 UTC
[jira] [Created] (KAFKA-5498) Connect validation API stops
returning recommendations for some fields after the right sequence of
requests
Ewen Cheslack-Postava created KAFKA-5498:
--------------------------------------------
Summary: Connect validation API stops returning recommendations for some fields after the right sequence of requests
Key: KAFKA-5498
URL: https://issues.apache.org/jira/browse/KAFKA-5498
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
Fix For: 0.11.0.0
If you issue the right sequence of requests against this API, it starts behaving differently, omitting certain fields (at a minimum recommended values, which is how I noticed this). If you start with
{code}
$ curl -X PUT -H "Content-Type: application/json" --data '{"connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector", "name": "file", "transforms": "foo"}' http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate | jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 5845 100 5730 100 115 36642 735 --:--:-- --:--:-- --:--:-- 36496
{
"name": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"error_count": 4,
"groups": [
"Common",
"Transforms",
"Transforms: foo"
],
"configs": [
{
"definition": {
"name": "name",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Globally unique name to use for this connector.",
"group": "Common",
"width": "MEDIUM",
"display_name": "Connector name",
"dependents": [],
"order": 1
},
"value": {
"name": "name",
"value": "file",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "connector.class",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Name or alias of the class for this connector. Must be a subclass of org.apache.kafka.connect.connector.Connector. If the connector is org.apache.kafka.connect.file.FileStreamSinkConnector, you can either specify this full name, or use \"FileStreamSink\" or \"FileStreamSinkConnector\" to make the configuration a bit shorter",
"group": "Common",
"width": "LONG",
"display_name": "Connector class",
"dependents": [],
"order": 2
},
"value": {
"name": "connector.class",
"value": "org.apache.kafka.connect.file.FileStreamSourceConnector",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "tasks.max",
"type": "INT",
"required": false,
"default_value": "1",
"importance": "HIGH",
"documentation": "Maximum number of tasks to use for this connector.",
"group": "Common",
"width": "SHORT",
"display_name": "Tasks max",
"dependents": [],
"order": 3
},
"value": {
"name": "tasks.max",
"value": "1",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "key.converter",
"type": "CLASS",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the keys in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.",
"group": "Common",
"width": "SHORT",
"display_name": "Key converter class",
"dependents": [],
"order": 4
},
"value": {
"name": "key.converter",
"value": null,
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "value.converter",
"type": "CLASS",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Converter class used to convert between Kafka Connect format and the serialized form that is written to Kafka. This controls the format of the values in messages written to or read from Kafka, and since this is independent of connectors it allows any connector to work with any serialization format. Examples of common formats include JSON and Avro.",
"group": "Common",
"width": "SHORT",
"display_name": "Value converter class",
"dependents": [],
"order": 5
},
"value": {
"name": "value.converter",
"value": null,
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "transforms",
"type": "LIST",
"required": false,
"default_value": null,
"importance": "LOW",
"documentation": "Aliases for the transformations to be applied to records.",
"group": "Transforms",
"width": "LONG",
"display_name": "Transforms",
"dependents": [],
"order": 6
},
"value": {
"name": "transforms",
"value": "foo",
"recommended_values": [],
"errors": [],
"visible": true
}
},
{
"definition": {
"name": "transforms.foo.type",
"type": "CLASS",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Class for the 'foo' transformation.",
"group": "Transforms: foo",
"width": "LONG",
"display_name": "Transformation type for foo",
"dependents": [],
"order": 0
},
"value": {
"name": "transforms.foo.type",
"value": null,
"recommended_values": [
"org.apache.kafka.connect.transforms.Cast$Key",
"org.apache.kafka.connect.transforms.Cast$Value",
"org.apache.kafka.connect.transforms.ExtractField$Key",
"org.apache.kafka.connect.transforms.ExtractField$Value",
"org.apache.kafka.connect.transforms.Flatten$Key",
"org.apache.kafka.connect.transforms.Flatten$Value",
"org.apache.kafka.connect.transforms.HoistField$Key",
"org.apache.kafka.connect.transforms.HoistField$Value",
"org.apache.kafka.connect.transforms.InsertField$Key",
"org.apache.kafka.connect.transforms.InsertField$Value",
"org.apache.kafka.connect.transforms.MaskField$Key",
"org.apache.kafka.connect.transforms.MaskField$Value",
"org.apache.kafka.connect.transforms.RegexRouter",
"org.apache.kafka.connect.transforms.ReplaceField$Key",
"org.apache.kafka.connect.transforms.ReplaceField$Value",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Key",
"org.apache.kafka.connect.transforms.SetSchemaMetadata$Value",
"org.apache.kafka.connect.transforms.TimestampConverter$Key",
"org.apache.kafka.connect.transforms.TimestampConverter$Value",
"org.apache.kafka.connect.transforms.TimestampRouter",
"org.apache.kafka.connect.transforms.ValueToKey"
],
"errors": [
"Missing required configuration \"transforms.foo.type\" which has no default value.",
"Invalid value null for configuration transforms.foo.type: Not a Transformation"
],
"visible": true
}
},
{
"definition": {
"name": "file",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "Source filename.",
"group": null,
"width": "NONE",
"display_name": "file",
"dependents": [],
"order": -1
},
"value": {
"name": "file",
"value": null,
"recommended_values": [],
"errors": [
"Missing required configuration \"file\" which has no default value."
],
"visible": true
}
},
{
"definition": {
"name": "topic",
"type": "STRING",
"required": true,
"default_value": null,
"importance": "HIGH",
"documentation": "The topic to publish data to",
"group": null,
"width": "NONE",
"display_name": "topic",
"dependents": [],
"order": -1
},
"value": {
"name": "topic",
"value": null,
"recommended_values": [],
"errors": [
"Missing required configuration \"topic\" which has no default value."
],
"visible": true
}
}
]
}
{code}
You can see that the recommended values for transformations are there. However, if you do
{code}
curl -X PUT -H "Content-Type: application/json" --data '{"connector.class": "org.apache.kafka.connect.file.FileStreamSourceConnector", "name": "file"}' http://localhost:8083/connector-plugins/org.apache.kafka.connect.file.FileStreamSourceConnector/config/validate | jq
{code}
before that command, the recommended values will not appear.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)