You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by skidder <gi...@git.apache.org> on 2016/07/12 02:39:46 UTC

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

GitHub user skidder opened a pull request:

    https://github.com/apache/flink/pull/2227

    [FLINK-4197] Allow Kinesis endpoint to be overridden via config

    I perform local testing of my application stack with Flink configured as a consumer on a Kinesis stream provided by Kinesalite, an implementation of Kinesis built on LevelDB. This requires me to override the AWS endpoint to refer to my local Kinesalite server rather than reference the real AWS endpoint. I'd like to add a configuration property to the Kinesis streaming connector that allows the AWS endpoint to be specified explicitly.
    
    This should be a fairly small change and provide a lot of flexibility to people looking to integrate Flink with Kinesis in a non-production setup.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/skidder/flink kinesis-url-override

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/2227.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2227
    
----
commit 0356141fbd826553cc206ac19dc4cfd734668c97
Author: Scott Kidder <sc...@mux.com>
Date:   2016-07-12T02:20:55Z

    [FLINK-4197] Allow Kinesis endpoint to be overridden via config

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70458352
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,41 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using non-AWS Kinesis Endpoints
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    +
    +To override the AWS endpoint, set the `CONFIG_AWS_ENDPOINT` property in the Flink configuration, in addition to the `CONFIG_AWS_REGION` required by Flink. Although the region is required, it will not be used to determine the AWS endpoint URL.
    --- End diff --
    
    Previous content in the documentation always have the `KinesisConfigConstants.*` prefix when mentioning config keys. Let's keep consistency with that.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by skidder <gi...@git.apache.org>.
Github user skidder commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    @tzulitai definitely, I'll add an explanation of the config setting in the docs shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70457990
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,41 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using non-AWS Kinesis Endpoints
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    --- End diff --
    
    We should be linking "kinesalite" to the project. Should we also capitalize "K"?
    
    However, I'm not sure if it's a good idea to refer to another public Github project in the documentation, even though Kinesalite is a widely adopted solution for local Kinesis testing. We'll need to have @rmetzger 's comment on this. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    Thank you for opening a PR for this @skidder ! I'll review the changes soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    @skidder I see, thank you for your explanation. If it's a familiar behavior (with respect to AWS / Kinesis users) to completely override the region when setting the endpoint regardless of the original value, then I think it's ok.
    
    Can you also add information about the overriding behaviour in the Kinesis connector documentation (can be found at `flink/docs/api/streaming/connectors/kinesis.md`)? I think it'll be good to have this explained.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by skidder <gi...@git.apache.org>.
Github user skidder commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70458869
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,41 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using non-AWS Kinesis Endpoints
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    --- End diff --
    
    Yes, thanks, I wasn't sure what the policy is for referencing other non-Apache projects.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    Looks good to merge.
    Waiting for the tests to pass, then merging (probably tomorrow).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/flink/pull/2227


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    Hi @skidder, I like the add, I think serious users will definitely need this.
    
    One thing I'm concerned about is how `CONFIG_AWS_REGION` and `CONFIG_AWS_ENDPOINT` should co-exist in the configuration properties, especially when we validate the configuration with `KinesisConfigUtil#validateConfiguration()`.
    The Kinesis consumer requires the user to specify `CONFIG_AWS_REGION`.
    I wonder if we need to check for conflicting `CONFIG_AWS_REGION` and `CONFIG_AWS_ENDPOINT` values, i.e.region is set to "us-east-1", but endpoint is set to "ap-northeast-1.amazonaws.com".
    
    Also, 
    
    What do you think?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70459536
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,41 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using Non-AWS Kinesis Endpoints for Testing
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    +
    +To override the AWS endpoint, set the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` property in the Flink configuration, in addition to the `KinesisConfigConstants.CONFIG_AWS_REGION` required by Flink. Although the region is required, it will not be used to determine the AWS endpoint URL.
    +
    +The following example shows how one might supply the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` configuration property:
    +
    +<div class="codetabs" markdown="1">
    +<div data-lang="java" markdown="1">
    +{% highlight java %}
    +Properties kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_ENDPOINT,
    +    "http://localhost:4567");
    +{% endhighlight %}
    +</div>
    +<div data-lang="scala" markdown="1">
    +{% highlight scala %}
    +val kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_ENDPOINT, "http://localhost:4567");
    --- End diff --
    
    nit: This config is written as a single line in the Scala example, but spans 3 lines in the Java example. Would be good to keep them align :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70478621
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,43 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using Non-AWS Kinesis Endpoints for Testing
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    +
    +To override the AWS endpoint, set the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` property in the Flink configuration, in addition to the `KinesisConfigConstants.CONFIG_AWS_REGION` required by Flink. Although the region is required, it will not be used to determine the AWS endpoint URL.
    +
    +The following example shows how one might supply the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` configuration property:
    +
    +<div class="codetabs" markdown="1">
    +<div data-lang="java" markdown="1">
    +{% highlight java %}
    +Properties kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_ENDPOINT,
    +    "http://localhost:4567");
    +{% endhighlight %}
    +</div>
    +<div data-lang="scala" markdown="1">
    +{% highlight scala %}
    +val kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_ENDPOINT,
    +    "http://localhost:4567");
    --- End diff --
    
    Mh, I'm undecided on this one. Its a lot of "code" for showing how to use one config property, but since its on the bottom of the page, regular user's are probably not distracted by this.
    I'd say lets keep it for now. If the kinesis documentation keeps growing, we can reconsider.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    @skidder Thank you for the quick responses & fixes!
    
    @rmetzger Can you also have a look through the changes & the comment on the Kinesalite referencing? Will also need help on the final merging, the changes LGTM, after comments are addressed. Thank you!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70462752
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,43 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using Non-AWS Kinesis Endpoints for Testing
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    +
    +To override the AWS endpoint, set the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` property in the Flink configuration, in addition to the `KinesisConfigConstants.CONFIG_AWS_REGION` required by Flink. Although the region is required, it will not be used to determine the AWS endpoint URL.
    +
    +The following example shows how one might supply the `KinesisConfigConstants.CONFIG_AWS_ENDPOINT` configuration property:
    +
    +<div class="codetabs" markdown="1">
    +<div data-lang="java" markdown="1">
    +{% highlight java %}
    +Properties kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_ENDPOINT,
    +    "http://localhost:4567");
    +{% endhighlight %}
    +</div>
    +<div data-lang="scala" markdown="1">
    +{% highlight scala %}
    +val kinesisProducerConfig = new Properties();
    +kinesisProducerConfig.put(KinesisConfigConstants.CONFIG_AWS_REGION, "us-east-1");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_ACCESSKEYID,
    +    "aws_access_key_id_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_CREDENTIALS_PROVIDER_BASIC_SECRETKEY,
    +    "aws_secret_key_here");
    +kinesisProducerConfig.put(
    +    KinesisConfigConstants.CONFIG_AWS_ENDPOINT,
    +    "http://localhost:4567");
    --- End diff --
    
    Depending on how we continue with the Kinesalite referencing, I'm not sure whether or not this example might be a bit too "_Kinesalite-specific_". Let's wait until we figure out the above before proceeding.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #2227: [FLINK-4197] Allow Kinesis endpoint to be overridden via ...

Posted by skidder <gi...@git.apache.org>.
Github user skidder commented on the issue:

    https://github.com/apache/flink/pull/2227
  
    Thanks for the comment, @tzulitai !
    
    I have not had any problems with testing Flink against a local Kinesalite instance where the Region (required by Flink) is set to `us-east-1` and the endpoint is set to an address on localhost. The `setRegion` function on the AWS Kinesis client calculates the endpoint URL from the region name and sets it on the client. The `setEndpoint` function is meant to allow for testing against non-AWS endpoints, or even new AWS endpoints in Regions unrecognized by an older version of the library.
    
    I think that requiring the AWS region, and optionally using the AWS endpoint if provided, will work well; this is consistent with the AWS client's own behavior.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70461744
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,43 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using Non-AWS Kinesis Endpoints for Testing
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    --- End diff --
    
    (original inline comment was marked outdated. Copying the comment again, just in case we miss it ;) )
    
    We should be linking "kinesalite" to the project. Should we also capitalize "K"?
    
    However, I'm not sure if it's a good idea to refer to another public Github project in the documentation, even though Kinesalite is a widely adopted solution for local Kinesis testing. We'll need to have @rmetzger 's comment on this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by tzulitai <gi...@git.apache.org>.
Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70457227
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,41 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using non-AWS Kinesis Endpoints
    --- End diff --
    
    "N"on-AWS. Also, I think it'd be good to let the title be "Using Non-AWS Kinesis Endpoints for Testing", to better point out the purpose of doing this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink pull request #2227: [FLINK-4197] Allow Kinesis endpoint to be overridd...

Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on a diff in the pull request:

    https://github.com/apache/flink/pull/2227#discussion_r70477574
  
    --- Diff: docs/apis/streaming/connectors/kinesis.md ---
    @@ -299,3 +299,43 @@ Otherwise, the returned stream name is used.
     Other optional configuration keys can be found in `KinesisConfigConstants`.
     		
     		
    +### Using Non-AWS Kinesis Endpoints for Testing
    +
    +It is sometimes desirable to have Flink operate as a consumer or producer against a non-AWS Kinesis endpoint such as kinesalite; this is especially useful when performing functional testing of a Flink application. The AWS endpoint that would normally be inferred by the AWS region set in the Flink configuration must be overriden via a configuration property.
    --- End diff --
    
    I don't think this is super critical to fix. Its easy to find the tool using a search engine, and its just an example.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---