From Indie IT Wiki
Jump to: navigation, search

S3 or Amazon Simple Storage Service provides a simple web-services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.


AWS Free Tier

Includes 5GB storage, 20,000 Get Requests, and 2,000 Put Requests with Amazon S3.



Standard 0.02 per GB
Infrequent Access 0.01 per GB
Glacier 0.005 per GB

Optimising Costs

Costs Example


S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing "buckets" and uploading, downloading and removing "objects" from these buckets. It runs on Linux and Mac.


CLI aws Install

This is the official AWS command line tool.

sudo -i
python --version
pip install awscli

BASH Completion -

CLI aws Usage

aws configure
aws help
aws s3 help
aws s3 ls
aws s3 sync /tmp/foo s3://bucketname/foo
aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 22 --cidr


1. To backup photos in your Syncthing directory (dryrun option added for testing)...

aws s3 sync --dryrun --exclude "*" --include "*201701*" /home/user/Syncthing/User/phone/photos/ s3://
# script to backup photos (taken the day before) to aws s3
YEAR=$( date +'%Y' -d "yesterday" )
MONTH=$( date +'%m' -d "yesterday" )
/usr/local/bin/aws s3 sync --exclude "*" --include "*${YEAR}${MONTH}*" /home/user/Syncthing/User/phone/photos/ s3://${YEAR}/${MONTH}/

2. To move objects from one bucket to another bucket, or same bucket but different folder...

aws s3 mv s3://source/file1.txt s3://destination/file2.txt

aws s3 mv s3://source/file1.txt s3://source/folder/file1.txt

aws --profile profile2 s3 mv --dryrun --recursive --exclude "*" --include "archive-nfs/201502*" s3://source/ s3://destination/archive-nfs/MailArchive/

3. To use a different profile (for different customers)...

nano ~/.aws/credentials

aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXXXXXXXXXX

aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXXXXXXXXXX
aws --profile customer2 s3 ls

Thanks -

4. Delete multiple files...

aws --profile customer2 s3 rm --dryrun --recursive --exclude "*" --include "messages" s3://bucket/folder/
(dryrun) delete: s3://bucket/folder/subfolder/messages

5. Make bucket...

aws s3 mb s3://mybucket --region eu-west-1

6. Create folder... (the key here is the forward slash / at the end)

aws s3api put-object --bucket test --key dir-test/

7. Size and Number of Files...

aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Official Guides

User Guide -

Reference -

CLI s3cmd Install

sudo -i
cd /root/misc
git clone
cd s3cmd
python install
s3cmd --version

Simple Backup Procedure With Retention Policy

Encrypted Incremental Backups with S3cmd

Install Error: No module named setuptools

If you receive the following error...

Traceback (most recent call last):
  File "", line 7, in <module>
    from setuptools import setup
ImportError: No module named setuptools

...then install the setuptools python module using pip...

sudo -i
cd /root/misc
pip install --upgrade setuptools


sudo -i
cd /root/misc/s3cmd
git pull
python install
s3cmd --version


s3cmd --configure

Tweak Settings

nano ~/.s3cfg
bucket_location = EU
host_bucket = %(bucket)

Create A Bucket

s3cmd mb s3://uniquename.subname.whatever

List Buckets

s3cmd ls

List Contents Of Buckets

s3cmd ls s3://uniquename.subname.whatever/

Create Directory

This is a bit strange but you have to upload a file to the whole folder tree that is not there. It will then create the folders and subfolders as part of the process.

s3cmd put /tmp/test.txt s3://uniquename.subname.whatever/folder/subfolder/test.txt

Upload Files (Test)

s3cmd put --recursive --dry-run ~/folder s3://uniquename.subname.whatever/

Upload Files

s3cmd put --recursive ~/folder s3://uniquename.subname.whatever/

Sync Files

s3cmd sync --verbose ~/folder s3://uniquename.subname.whatever/

Example: Backup Dovecot Emails Script

cd /var/vmail/ && \
/bin/tar -cpf && \
/usr/local/bin/s3cmd --quiet put /var/vmail/ s3:// && \
/usr/local/bin/s3cmd ls -H s3://

Calculate Size of Bucket


Log in > S3 > Click on Bucket > Select All (tickboxes) > More > Get Size


aws s3 ls --summarize --human-readable --recursive s3://bucket/


s3cmd du s3://bucket/ --human-readable

Restrict Access From An IP Address

Android App

Bucket Policy Examples

Restrict To Single Bucket

   "Statement": [
           "Action": "s3:ListAllMyBuckets",
           "Effect": "Allow",
           "Resource": "arn:aws:s3:::*"
           "Action": "s3:*",
           "Effect": "Allow",
           "Resource": "arn:aws:s3:::mybucketname"


 "Statement": [
     "Effect": "Allow",
     "Action": [
     "Resource": "arn:aws:s3:::mybucketname",
     "Condition": {}
     "Effect": "Allow",
     "Action": [
     "Resource": "arn:aws:s3:::mybucketname/*",
     "Condition": {}
     "Effect": "Allow",
     "Action": "s3:ListAllMyBuckets",
     "Resource": "*",
     "Condition": {}

Event Notifications


Edit your Topic Policy to allow S3 to publish events to SNS...

"Version": "2008-10-17",
"Id": "example-ID",
"Statement": [
  "Sid": "example-statement-ID",
  "Effect": "Allow",
  "Principal": {
  "Action": [
  "Resource": "Topic-ARN",
  "Condition": {
     "ArnLike": {          
     "aws:SourceArn": "arn:aws:s3:*:*"    

Lifecycle Storage Management with Glacier

Because Amazon S3 maintains the mapping between your user-defined object name and Amazon Glacier’s system-defined identifier, Amazon S3 objects that are stored using the Amazon Glacier option are only accessible through the Amazon S3 APIs or the Amazon S3 Management Console.

To put this in slightly simpler terms, S3 doesn't create Glacier archives that you own or can manipulate. S3 creates Glacier archives that S3 owns and manages.

Your only interface to these objects is through S3, which makes requests to Glacier on your behalf. So, for your questions, the answer for each one is essentially the same:

It doesn't matter. The archives are managed by S3 and are not user-accessible via the Glacier API or console.


AWS S3 Lifecycle Storage Management with Glacier


WARNING: Redirected To

Replace the bucket_host in the .s3cfg file with the one from the warning.


host_bucket = %(bucket)

Thanks to


CloudBerry Backup

Cloudberry - Simple backup software that stores the data in its simple folder structure.

It has a web based interface available at port 43210 -


Cyberduck - Mounts the S3 storage in your desktop (Windows or Mac) file browser.

S3 Sync (Windows)

Sprightly Soft S3Sync

$29.99 USD

Bonkey (The Backup Monkey) (Mac & Windows)

Home Page.

Duplicati (Crossplatform)

Once installed you Duplicati will open a web interface:

ARQ Backup (MAC)