AWS S3

From Indie IT Wiki
Jump to: navigation, search

S3 or Amazon Simple Storage Service provides a simple web-services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure that Amazon uses to run its own global network of web sites. The service aims to maximize benefits of scale and to pass those benefits on to developers.

INFO:

AWS Free Tier

Includes 5GB storage, 20,000 Get Requests, and 2,000 Put Requests with Amazon S3.

http://aws.amazon.com/s3/

Introduction

http://amzn.to/1rlFqoH

Pricing

http://aws.amazon.com/s3/pricing/

http://calculator.s3.amazonaws.com/index.html

Optimising Costs

http://www.wikihow.com/Optimize-Your-Amazon-S3-Costs

Costs Example

https://stackoverflow.com/a/43988317/3163050

s3cmd

S3cmd is a tool for managing objects in Amazon S3 storage. It allows for making and removing "buckets" and uploading, downloading and removing "objects" from these buckets. It runs on Linux and Mac.

http://s3tools.org

HOWTO:

CLI aws Install

This is the official AWS command line tool.

sudo -i
python --version
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip install awscli

BASH Completion - http://docs.aws.amazon.com/cli/latest/userguide/cli-command-completion.html

CLI aws Usage

aws configure
aws help
aws s3 help
aws s3 ls
aws s3 sync /tmp/foo s3://bucketname/foo
aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 22 --cidr xxx.xxx.xx.xx/32

CLI_Examples

1. To backup photos in your Syncthing directory (dryrun option added for testing)...

aws s3 sync --dryrun --exclude "*" --include "*201701*" /home/user/Syncthing/User/phone/photos/ s3://user.name.bucket/Photos/2017/01/
#!/bin/bash
#
# script to backup photos (taken the day before) to aws s3
#
YEAR=$( date +'%Y' -d "yesterday" )
MONTH=$( date +'%m' -d "yesterday" )
/usr/local/bin/aws s3 sync --exclude "*" --include "*${YEAR}${MONTH}*" /home/user/Syncthing/User/phone/photos/ s3://user.name.bucket/Photos/${YEAR}/${MONTH}/
exit

2. To move objects from one bucket to another bucket, or same bucket but different folder...

aws s3 mv s3://source/file1.txt s3://destination/file2.txt

aws s3 mv s3://source/file1.txt s3://source/folder/file1.txt

aws --profile profile2 s3 mv --dryrun --recursive --exclude "*" --include "archive-nfs/201502*" s3://source/ s3://destination/archive-nfs/MailArchive/

3. To use a different profile (for different customers)...

nano ~/.aws/credentials

[default]
aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXXXXXXXXXX

[customer2]
aws_access_key_id = XXXXXX
aws_secret_access_key = XXXXXXXXXXXXX
aws --profile customer2 s3 ls

Thanks - http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-multiple-profiles

4. Delete multiple files...

aws --profile customer2 s3 rm --dryrun --recursive --exclude "*" --include "messages" s3://bucket/folder/
(dryrun) delete: s3://bucket/folder/subfolder/messages

5. Make bucket...

aws s3 mb s3://mybucket --region eu-west-1

6. Create folder... (the key here is the forward slash / at the end)

aws s3api put-object --bucket test --key dir-test/

7. Size and Number of Files...

aws s3api list-objects --bucket BUCKETNAME --output json --query "[sum(Contents[].Size), length(Contents[])]"

Official Guides

User Guide - http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html

Reference - http://docs.aws.amazon.com/cli/latest/index.html

CLI s3cmd Install

sudo -i
cd /root/misc
git clone https://github.com/s3tools/s3cmd.git
cd s3cmd
python setup.py install
s3cmd --version
exit

Simple Backup Procedure With Retention Policy

https://cloudacademy.com/blog/data-backup-s3cmd/

Encrypted Incremental Backups with S3cmd

https://www.bentasker.co.uk/documentation/linux/285-implementing-encrypted-incremental-backups-with-s3cmd

Install Error: No module named setuptools

If you receive the following error...

Traceback (most recent call last):
  File "setup.py", line 7, in <module>
    from setuptools import setup
ImportError: No module named setuptools

...then install the setuptools python module using pip...

sudo -i
cd /root/misc
wget https://bootstrap.pypa.io/get-pip.py
python get-pip.py
pip install --upgrade setuptools

Update

sudo -i
cd /root/misc/s3cmd
git pull
python setup.py install
s3cmd --version
exit

Configure

s3cmd --configure

Tweak Settings

nano ~/.s3cfg
bucket_location = EU
host_bucket = %(bucket)s.s3-external-3.amazonaws.com

Create A Bucket

s3cmd mb s3://uniquename.subname.whatever

List Buckets

s3cmd ls

List Contents Of Buckets

s3cmd ls s3://uniquename.subname.whatever/

Create Directory

This is a bit strange but you have to upload a file to the whole folder tree that is not there. It will then create the folders and subfolders as part of the process.

s3cmd put /tmp/test.txt s3://uniquename.subname.whatever/folder/subfolder/test.txt

Upload Files (Test)

s3cmd put --recursive --dry-run ~/folder s3://uniquename.subname.whatever/

Upload Files

s3cmd put --recursive ~/folder s3://uniquename.subname.whatever/

Sync Files

s3cmd sync --verbose ~/folder s3://uniquename.subname.whatever/

Example: Backup Dovecot Emails Script

#!/bin/bash
cd /var/vmail/ && \
/bin/tar -cpf domain.co.uk.tar domain.co.uk && \
/usr/local/bin/s3cmd --quiet put /var/vmail/domain.co.uk.tar s3://domain.co.uk.aws2/var/vmail/ && \
/usr/local/bin/s3cmd ls -H s3://domain.co.uk.aws2/var/vmail/domain.co.uk.tar

Restrict Access From An IP Address

https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3

http://s3tools.org/kb/item10.htm

Android App

http://www.lysesoft.com/products/s3anywhere/

Bucket Policy Examples

http://s3browser.com/working-with-amazon-s3-bucket-policies.php

Restrict To Single Bucket

{
   "Statement": [
       {
           "Action": "s3:ListAllMyBuckets",
           "Effect": "Allow",
           "Resource": "arn:aws:s3:::*"
       }
       {
           "Action": "s3:*",
           "Effect": "Allow",
           "Resource": "arn:aws:s3:::mybucketname"
       }
   ]
}

or

{
 "Statement": [
   {
     "Effect": "Allow",
     "Action": [
       "s3:ListBucket",
       "s3:GetBucketLocation",
       "s3:ListBucketMultipartUploads"
     ],
     "Resource": "arn:aws:s3:::mybucketname",
     "Condition": {}
   },
   {
     "Effect": "Allow",
     "Action": [
       "s3:AbortMultipartUpload",
       "s3:DeleteObject",
       "s3:DeleteObjectVersion",
       "s3:GetObject",
       "s3:GetObjectAcl",
       "s3:GetObjectVersion",
       "s3:GetObjectVersionAcl",
       "s3:PutObject",
       "s3:PutObjectAcl",
       "s3:PutObjectAclVersion"
     ],
     "Resource": "arn:aws:s3:::mybucketname/*",
     "Condition": {}
   },
   {
     "Effect": "Allow",
     "Action": "s3:ListAllMyBuckets",
     "Resource": "*",
     "Condition": {}
   }
 ]
}

Event Notifications

SNS

Edit your Topic Policy to allow S3 to publish events to SNS...

{
"Version": "2008-10-17",
"Id": "example-ID",
"Statement": [
 {
  "Sid": "example-statement-ID",
  "Effect": "Allow",
  "Principal": {
   "AWS":"*"  
  },
  "Action": [
   "SNS:Publish"
  ],
  "Resource": "Topic-ARN",
  "Condition": {
     "ArnLike": {          
     "aws:SourceArn": "arn:aws:s3:*:*:bucket.name"    
   }
  }
 }
]
}

HOWTO: FIX:

WARNING: Redirected To

Replace the bucket_host in the .s3cfg file with the one from the warning.

~/.s3cfg

host_bucket = %(bucket)s.s3-external-3.amazonaws.com

Thanks to ServerFault.com.

THIRD PARTY SOFTWARE

CloudBerry Backup

Cloudberry - Simple backup software that stores the data in its simple folder structure.

Cyberduck

Cyberduck - Mounts the S3 storage in your desktop (Windows or Mac) file browser.

S3 Sync (Windows)

Sprightly Soft S3Sync

$29.99 USD

Bonkey (The Backup Monkey) (Mac & Windows)

Home Page.

Duplicati (Crossplatform)

http://www.duplicati.com/

Once installed you Duplicati will open a web interface:

http://127.0.0.1:8200/ngax/index.html

ARQ Backup (MAC)

https://www.arqbackup.com/