You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Steve Loughran (Jira)" <ji...@apache.org> on 2021/07/01 10:29:00 UTC
[jira] [Commented] (HADOOP-17784) hadoop-aws test bucket woul be
deleted on Jul 1, 2021
[ https://issues.apache.org/jira/browse/HADOOP-17784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17372617#comment-17372617 ]
Steve Loughran commented on HADOOP-17784:
-----------------------------------------
Uh oh. This is our critical public dataset as it gives us CSV files with no setup delays and AWS to pay the download charges.
We use it for testing
* random IO seek policy performance
* list performance with large directories
* S3 select.
This is going to be a disaster. We will have to see if we can find some other public store with what we need (a large CSV file that's free to download)
There's always been options to override the path to the CSV file, which was done to support private S3 store testing. We'll need to verify that switching to this will allow old builds to keep testing happily. Some test suites do skip if the target CSV file != that one in the constants (Region checking etc)
> hadoop-aws test bucket woul be deleted on Jul 1, 2021
> -----------------------------------------------------
>
> Key: HADOOP-17784
> URL: https://issues.apache.org/jira/browse/HADOOP-17784
> Project: Hadoop Common
> Issue Type: Test
> Components: fs/s3, test
> Reporter: Leona Yoda
> Priority: Major
>
> I found an anouncement that landsat-pds buket will be deleted on July 1, 2021
> (https://registry.opendata.aws/landsat-8/)
> and I think this bucket is used in th test of hadoop-aws module use
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/S3ATestConstants.java#L93]
>
> At this time I can access the bucket but we might have to change the test bucket in someday.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org