[jira] [Closed] (FLINK-25107) GlueSchemaRegistryAvroKinesisITCase and GlueSchemaRegistryJsonKinesisITCase are skipped on AzureCI but fail when enabled

["Danny Cranmer (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-25107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Danny Cranmer closed FLINK-25107.
---------------------------------
    Resolution: Fixed

> GlueSchemaRegistryAvroKinesisITCase and GlueSchemaRegistryJsonKinesisITCase are skipped on AzureCI but fail when enabled
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-25107
>                 URL: https://issues.apache.org/jira/browse/FLINK-25107
>             Project: Flink
>          Issue Type: Bug
>          Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>    Affects Versions: 1.15.0
>            Reporter: Matthias Pohl
>            Assignee: Danny Cranmer
>            Priority: Blocker
>              Labels: pull-request-available, test-stability
>             Fix For: 1.15.0
>
>
> [GlueSchemaRegistryAvroKinesisITCase|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-glue-schema-registry-avro-test/src/test/java/org/apache/flink/glue/schema/registry/test/GlueSchemaRegistryAvroKinesisITCase.java] and [GlueSchemaRegistryJsonKinesisITCase|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-glue-schema-registry-json-test/src/test/java/org/apache/flink/glue/schema/registry/test/json/GlueSchemaRegistryJsonKinesisITCase.java], are not executed on Azure CI runs, because the access key and secret key env variables are not present, see e.g. [this run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26986&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=15852].
> Then, during recent testing on Github Actions, we noticed that the tests don't work even if the env variables are present because AWS expects different variable names (it expects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). See [this run|https://github.com/ververica/flink/runs/4301833493?check_suite_focus=true#step:13:17885].
> {code:java}
> Nov 23 18:40:46 Caused by: software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(): Profile file contained no credentials for profile 'default': ProfileFile(profiles=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): The requested metadata is not found at http://169.254.169.254/latest/meta-data/iam/security-credentials/]
> {code}
> Finally, even with correct env variable naming, the test still fails because of an assertion error. It looks like only the first record ever makes it to the results. See [this run|https://github.com/ververica/flink/runs/4315084463?check_suite_focus=true#step:13:5317] for the error, also posted here for convenience:
> {code:java}
> Nov 24 18:00:55 java.lang.AssertionError: Results received from 'gsr_json_output_stream': [JsonDataWithSchema(schema={"$id":"https://example.com/address.schema.json","$schema":"http://json-schema.org/draft-07/schema#","type":"object","properties":{"f1":{"type":"string"},"f2":{"type":"integer","maximum":10000}}}, payload={"f1":"olympic","f2":2020})] expected:<8> but was:<1>
> {code}
> FYI: [~Nicolaus Weidner]  investigated this issue and provided these details



--
This message was sent by Atlassian Jira
(v8.20.1#820001)