[jira] [Resolved] (HADOOP-16644) Retrive modtime of PUT file from store, via response or HEAD

["Steve Loughran (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HADOOP-16644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Steve Loughran resolved HADOOP-16644.
-------------------------------------
    Resolution: Won't Fix

Update: put result doesn't include timestamp. unless we do a HEAD every time, this is in trouble.

I'm going to close the patch as is. If we were to revisit this it would be needed for every file and I'd make an option for some special cases (Yarn job submission) where timestamp mismatch is a blocker

> Retrive modtime of PUT file from store, via response or HEAD
> ------------------------------------------------------------
>
>                 Key: HADOOP-16644
>                 URL: https://issues.apache.org/jira/browse/HADOOP-16644
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3, test
>    Affects Versions: 3.3.0
>         Environment: -Dparallel-tests -DtestsThreadCount=8 -Dfailsafe.runOrder=balanced -Ds3guard -Ddynamo -Dscale
> h2. Hypothesis:
> the timestamp of the source file is being picked up from S3Guard, but when the NM does a getFileStatus call, a HEAD check is made -and this (due to the overloaded test system) is out of sync with the listing. S3Guard is updated, the corrected date returned and the localisation fails.
>            Reporter: Steve Loughran
>            Priority: Major
>
> Terasort of directory committer failing in resource localisaton -the partitions.lst file has a different TS from that expected
> Happens under loaded integration tests (threads = 8; not standalone); non-auth s3guard
> {code}
> 2019-10-08 11:50:29,774 [IPC Server handler 4 on 55983] WARN  localizer.ResourceLocalizationService (ResourceLocalizationService.java:processHeartbeat(1150)) - { s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst, 1570531828143, FILE, null } failed: Resource s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst changed on src filesystem (expected 1570531828143, was 1570531828000
> java.io.IOException: Resource s3a://hwdev-steve-ireland-new/terasort-directory/sortout/_partition.lst changed on src filesystem (expected 1570531828143, was 1570531828000
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org