Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Here is a good one: deleting billions of objects can be expensive if you call delete APIs.

However you can set a wildcard or bucket wide object expiry of time=now for free. You’ll immediately stop being charged for storage, and AWS will manage making sure everything is deleted.



Nit: the delete call is free, it’s the list call to get the objects that costs money. In theory if you know what objects you have from another source it’s free.


> You’ll immediately stop being charged for storage

The effect of lifecycle rules is not immediate: they get applied in a once-per-day batch job, so the removal is not immediate.


That's true but OP clearly knows that already. You stop getting charged for storage as soon as the object is marked for expiration, not when the object is finally removed at AWS's leisure. You can check the expiration status of objects in the metadata panel of the S3 web console.

https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...

> There may be a delay between the expiration date and the date at which Amazon S3 removes an object. You are not charged for expiration or the storage time associated with an object that has expired.


Because AWS gets to choose when the actual deletes happen. The metadata is marked as object deleted but AWS can process the delete off peak. It also avoids the s3 api server being hammered with qps


An explicit delete operation can mark metadata the same way.

The real difference is likely more along the lines of LSM compaction, with expiry they likely use a moral equivalent of https://github.com/facebook/rocksdb/wiki/Compaction-Filter to actually do the deletion.


That’s called a delete marker in s3. Less IO than a true delete but also still can lead to high qps to s3 api server. https://docs.aws.amazon.com/AmazonS3/latest/userguide/Delete...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: