[jira] [Created] (FLINK-29311) Unable to use http for Kinesis consumer when testing locally

["Nicolas Pocock (Jira)" <ji...@apache.org>]

Nicolas Pocock created FLINK-29311:
--------------------------------------

             Summary: Unable to use http for Kinesis consumer when testing locally
                 Key: FLINK-29311
                 URL: https://issues.apache.org/jira/browse/FLINK-29311
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kinesis, Stateful Functions
    Affects Versions: 1.15.2
         Environment: Here is my Kinesis module code

 

kind: io.statefun.kinesis.v1/ingress
spec:
  id: com.xia/sentiment
  awsRegion:
    id: us-east-1
    type: custom-endpoint
    customEndpoint: https://localstack:4566
  awsCredentials:
    type: basic
    id: us-east-1
    accessKeyId: key
    secretAccessKey: secret
  startupPosition:
    type: latest
  streams:
    - stream: customer-details
      valueType: xia/SentimentEvent
      targets:
        - xia/sentiment
  clientConfigProperties:
    - SocketTimeout: 9999
    - MaxConnections: 15

And I am using  the latest apache/statefun Docker image within docker compose
            Reporter: Nicolas Pocock


I am attempting to run the StateFun docker image locally in a docker-compose stack. I have Kinesis running using Localstack and want to use it as a consumer. 

The documentation says to use a custom endpoint you need to do:

awsCredentials:
  type: custom-endpoint
  endpoint: https://localhost:4567
  id: us-west-1

The above errors as the customEndpoint property is on the AWS region instead so it needs to be:

  awsRegion:
    id: us-east-1
    type: custom-endpoint
    customEndpoint: [http://localstack:4566|http://localstack:4566/]

However, I am unable to use HTTP as it complains that:

Caused by: java.lang.IllegalArgumentException: Invalid service endpoint url: http://localstack:4566; Only custom service endpoints using HTTPS are supported

I tried using HTTPS but it errors with a certificate issue. 

Some of the docs for the Kinesis connectors show HTTP being used with a custom endpoint

[https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#using-custom-kinesis-endpoints]

It would be nice if we can use HTTP when testing locally



--
This message was sent by Atlassian Jira
(v8.20.10#820010)