[GitHub] [druid] sascha-coenen edited a comment on issue #9411: S3 InputSource issue when using prefix mode if a directory contains _SUCCESS files

[GitBox <gi...@apache.org>]

sascha-coenen edited a comment on issue #9411: S3 InputSource issue when using prefix mode if a directory contains  _SUCCESS files
URL: https://github.com/apache/druid/issues/9411#issuecomment-591695099
 
 
   > Are you saying that the data files did not get ingested?
   
   yes. the ingestion fails as long as we have the _SUCCESS file in the folder. Then I deleted  the _SUCCESS file and the same ingestion spec resubmitted a second time was now working. At least the first phase was running successfully.
   
   WIth Druid 0.16.0 we had some successful attempts at using index_parallel. With 0.17.0 the second phase immediately fails. Currently we are clueless why, but as far as this report scope is concerned, it is sufficient that the first phase completes successfully. I can see in the logs that the files got picked up and processed and that there were as many slots used as there are files in the s3 location.
   
   When the _SUCCESS file is present, only one slot will be used, so it doesn't get to the point that the other files would also be scheduled. Perhaps I'm wrong. I can double check tomorrow to be sure. But now I know that there is no dedicated code to deal with shadowed files so I know to remove them for now.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org