How to delete lot of objects named with common prefix from s3 bucket?
I have files in s3 bucket, and their names have the following format:
username#file_id#...How to remove all john#doe#* items, without listing them? There are thousands of them, so when user request my app to delete all of them, he has to wait.
There's no way to tell S3 to delete all files that meet a specific criteria - you have to delete one key at a time.
Most client libraries offer a way to filter and paginate such that you'd only list the files you need to delete and you can provide a status update. For an example, Boto's bucket listing accepts prefix as one of the parameters.
For anyone who stumbles upon this now, you can create a lifecycle rule to either delete or set expiration of files with a certain prefix.
I have mistakenly create logging files in same bucket and there are like tons of log file in my bucket. Luckily I came across a nodejs util node-s3-utils and it save my day!
example of delete files with foo/ prefix, having extension .txt $ s3utils files delete -c ./.s3-credentials.json -p foo/ -r 'foo\/(\w)+\.txt'