Have you tried setting a redirection rule on your bucket so that when the 500 error occurs, S3 will automatically retry the request? You can set a redirection rule in the S3 console, and I think the following rule might work:
This will redirect all 500s back to the same location, effectively re-trying the request. This should cover the random 500 case but I'm not sure that it will work 100% of the time though.
I'm interested by this solution too. I will try to take a look and test it over the weekend.
I think that this solution can only be an improvement though.
Have you tried setting a redirection rule on your bucket so that when the 500 error occurs, S3 will automatically retry the request? You can set a redirection rule in the S3 console, and I think the following rule might work:
<RoutingRules> <RoutingRule> <Condition> <HttpErrorCodeReturnedEquals>500</HttpErrorCodeReturnedEquals> </Condition> <Redirect> <HttpRedirectCode>301</HttpRedirectCode> </Redirect> </RoutingRule> </RoutingRules>
This will redirect all 500s back to the same location, effectively re-trying the request. This should cover the random 500 case but I'm not sure that it will work 100% of the time though.