Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

The one that caught me a couple of weeks ago is multipart uploads have a minimum initial chunk size of 5 MiBs (https://docs.aws.amazon.com/AmazonS3/latest/userguide/qfacts...). I built a streaming CSV post-processing pipeline in Elixir that uses Stream.transform (https://hexdocs.pm/elixir/Stream.html#transform/3) to modify and inject columns. The Elixir AWS and CSV modules handle streaming data in but the AWS module throws an error (from S3) if you stream "out" that totals less than 5 MiBs as is uses multi-part uploads which made me sad.


The last part can be any size, so with a few tweaks to the streaming code you should be fine. Ready-made AWS SDKs handle this (chunking) for you. Truth be told, the multupart upload on GCP is even worse :/




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: