You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "cchenaxchen (Jira)" <ji...@apache.org> on 2022/02/24 08:36:00 UTC
[jira] [Assigned] (HDDS-6356) EC: the offset is less than writeoffset
[ https://issues.apache.org/jira/browse/HDDS-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
cchenaxchen reassigned HDDS-6356:
---------------------------------
Assignee: cchenaxchen
> EC: the offset is less than writeoffset
> ---------------------------------------
>
> Key: HDDS-6356
> URL: https://issues.apache.org/jira/browse/HDDS-6356
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: EC
> Reporter: chen chao
> Assignee: cchenaxchen
> Priority: Major
>
> I use the lastest code
> I run the mapreduce
> bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.1-tests.jar TestDFSIO -Dtest.build.data=o3fs://defaultbucket.vol1/dfsio/test2 -write -nrFiles 3 -size 10000MB
>
> Error: java.lang.IllegalArgumentException
> at org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
> at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:532)
> at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
>
> when I add some logs
> so the offset is less than writeOffset
> and the stripeSize is equal numDataBlks * ecChunkSize, it is a problem
> 2022-02-23 16:16:01,347 INFO [main] org.apache.hadoop.ozone.client.io.KeyOutputStream: name = main lastStripeSize is 3145728 writeOffset is 4838000000 offset is 4834983936
> 2022-02-23 16:16:01,349 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.IllegalArgumentException
> at org.apache.hadoop.ozone.shaded.com.google.common.base.Preconditions.checkArgument(Preconditions.java:130)
> at org.apache.hadoop.ozone.client.io.ECKeyOutputStream.close(ECKeyOutputStream.java:550)
> at org.apache.hadoop.fs.ozone.OzoneFSOutputStream.close(OzoneFSOutputStream.java:56)
> at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:72)
> at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:101)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:136)
> at org.apache.hadoop.fs.IOMapperBase.map(IOMapperBase.java:37)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org