AWS S3

S3 or Amazon Simple Storage Service provides a simple web-services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

AWS Free Tier
Includes 5GB storage, 20,000 Get Requests, and 2,000 Put Requests with Amazon S3.

http://aws.amazon.com/s3/

Introduction
http://amzn.to/1rlFqoH

Pricing
http://aws.amazon.com/s3/pricing/

http://calculator.s3.amazonaws.com/index.html

Optimising Costs
http://www.wikihow.com/Optimize-Your-Amazon-S3-Costs

s3cmd
S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing "buckets" and uploading, downloading and removing "objects" from these buckets. It runs on Linux and Mac.

http://s3tools.org

CLI aws Install
This is the official AWS command line tool.

sudo -i python --version wget https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install awscli

BASH Completion - http://docs.aws.amazon.com/cli/latest/userguide/cli-command-completion.html

CLI aws Usage
aws configure

aws help

aws s3 help

aws s3 ls

aws s3 sync /tmp/foo s3://bucketname/foo

aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 22 --cidr xxx.xxx.xx.xx/32

CLI_Examples
1. To backup photos in your Syncthing directory (dryrun option added for testing)...

aws s3 sync --dryrun --exclude "*" --include "*201701*" /home/user/Syncthing/User/phone/photos/ s3://user.name.bucket/Photos/2017/01/

# # YEAR=$( date +'%Y' -d "yesterday" ) MONTH=$( date +'%m' -d "yesterday" ) /usr/local/bin/aws s3 sync --exclude "*" --include "*${YEAR}${MONTH}*" /home/user/Syncthing/User/phone/photos/ s3://user.name.bucket/Photos/${YEAR}/${MONTH}/ exit
 * 1) !/bin/bash
 * 1) script to backup photos (taken the day before) to aws s3

2. To move objects from one bucket to another bucket, or same bucket but different folder...

aws s3 mv s3://source/file1.txt s3://destination/file2.txt aws s3 mv s3://source/file1.txt s3://source/folder/file1.txt aws --profile profile2 s3 mv --dryrun --recursive --exclude "*" --include "archive-nfs/201502*" s3://source/ s3://destination/archive-nfs/MailArchive/

3. To use a different profile (for different customers)...

nano ~/.aws/credentials [default] aws_access_key_id = XXXXXX aws_secret_access_key = XXXXXXXXXXXXX [customer2] aws_access_key_id = XXXXXX aws_secret_access_key = XXXXXXXXXXXXX

aws --profile customer2 s3 ls

Thanks - http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

4. Delete multiple files...

aws --profile customer2 s3 rm --dryrun --recursive --exclude "*" --include "messages" s3://bucket/folder/ (dryrun) delete: s3://bucket/folder/subfolder/messages

5. Make bucket...

aws s3 mb s3://mybucket --region eu-west-1

6. Create folder... (the key here is the forward slash / at the end)

aws s3api put-object --bucket test --key dir-test/

7. Size and Number of Files...

aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Official Guides

User Guide - http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

Reference - http://docs.aws.amazon.com/cli/latest/index.html

CLI s3cmd Install
sudo -i cd /root/misc git clone https://github.com/s3tools/s3cmd.git cd s3cmd python setup.py install s3cmd --version exit

Install Error: No module named setuptools
If you receive the following error...

Traceback (most recent call last): File "setup.py", line 7, in     from setuptools import setup ImportError: No module named setuptools

...then install the setuptools python module using pip...

sudo -i cd /root/misc wget https://bootstrap.pypa.io/get-pip.py python get-pip.py pip install --upgrade setuptools

Update
sudo -i cd /root/misc/s3cmd git pull python setup.py install s3cmd --version exit

Configure
s3cmd --configure

Tweak Settings
nano ~/.s3cfg

bucket_location = EU host_bucket = %(bucket)s.s3-external-3.amazonaws.com

Create A Bucket
s3cmd mb s3://uniquename.subname.whatever

List Buckets
s3cmd ls

List Contents Of Buckets
s3cmd ls s3://uniquename.subname.whatever/

Create Directory
This is a bit strange but you have to upload a file to the whole folder tree that is not there. It will then create the folders and subfolders as part of the process.

s3cmd put /tmp/test.txt s3://uniquename.subname.whatever/folder/subfolder/test.txt

Upload Files (Test)
s3cmd put --recursive --dry-run ~/folder s3://uniquename.subname.whatever/

Upload Files
s3cmd put --recursive ~/folder s3://uniquename.subname.whatever/

Sync Files
s3cmd sync --verbose ~/folder s3://uniquename.subname.whatever/

Example: Backup Dovecot Emails Script
cd /var/vmail/ && \ /bin/tar -cpf domain.co.uk.tar domain.co.uk && \ /usr/local/bin/s3cmd --quiet put /var/vmail/domain.co.uk.tar s3://domain.co.uk.aws2/var/vmail/ && \ /usr/local/bin/s3cmd ls -H s3://domain.co.uk.aws2/var/vmail/domain.co.uk.tar
 * 1) !/bin/bash

Restrict Access From An IP Address
https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3

http://s3tools.org/kb/item10.htm

Android App
http://www.lysesoft.com/products/s3anywhere/

Bucket Policy Examples
http://s3browser.com/working-with-amazon-s3-bucket-policies.php

Restrict To Single Bucket
{   "Statement": [ {           "Action": "s3:ListAllMyBuckets", "Effect": "Allow", "Resource": "arn:aws:s3:::*" }       {            "Action": "s3:*", "Effect": "Allow", "Resource": "arn:aws:s3:::mybucketname" }   ] }

or

{ "Statement": [ {     "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:ListBucketMultipartUploads" ],     "Resource": "arn:aws:s3:::mybucketname", "Condition": {} },   {      "Effect": "Allow", "Action": [ "s3:AbortMultipartUpload", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:GetObject", "s3:GetObjectAcl", "s3:GetObjectVersion", "s3:GetObjectVersionAcl", "s3:PutObject", "s3:PutObjectAcl", "s3:PutObjectAclVersion" ],     "Resource": "arn:aws:s3:::mybucketname/*", "Condition": {} },   {      "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*", "Condition": {} } ] }

SNS
Edit your Topic Policy to allow S3 to publish events to SNS...

{ "Version": "2008-10-17", "Id": "example-ID", "Statement": [ {  "Sid": "example-statement-ID", "Effect": "Allow", "Principal": { "AWS":"*" },  "Action": [ "SNS:Publish" ],  "Resource": "Topic-ARN", "Condition": { "ArnLike": { "aws:SourceArn": "arn:aws:s3:*:*:bucket.name" }  }  } ] }

WARNING: Redirected To
Replace the bucket_host in the .s3cfg file with the one from the warning.

~/.s3cfg host_bucket = %(bucket)s.s3-external-3.amazonaws.com

Thanks to ServerFault.com.

CloudBerry Backup
Cloudberry - Simple backup software that stores the data in its simple folder structure.

Cyberduck
Cyberduck - Mounts the S3 storage in your desktop (Windows or Mac) file browser.

S3 Sync (Windows)
Sprightly Soft S3Sync

$29.99 USD

Bonkey (The Backup Monkey) (Mac & Windows)
Home Page.

Duplicati (Crossplatform)
http://www.duplicati.com/

Once installed you Duplicati will open a web interface:

http://127.0.0.1:8200/ngax/index.html

ARQ Backup (MAC)
https://www.arqbackup.com/