AWS EC2

From Indie IT Wiki

INFO:

EC2 from Amazon stands for "Elastic Compute Cloud", and is a core component of their cloud based computing services collectively know as Amazon Web Services (AWS).

NOTE: In the following tutorial replace <yourdomain.com> with your own domain name.

Volumes

Volumes are hard disk drives. They can be attached to a running virtual machine (just like plugging the cables in), and then have to be partitioned, formatted and mounted.

EBS

EBS is Elastic Block Storage and basically means that Amazon use a whole bunch of drives stored over several servers and then put those together to make your disk or volume. After a volume is attached to an instance, you can use it like any other physical hard drive. EBS volumes persist independently from the running life of an EC2 instance. This is just like having a physical hard drive, unplugging it and keeping the data on it. When you create an EBS volume in an Availability Zone, it is automatically replicated within that zone to prevent data loss due to failure of any single hardware component. After you create a volume, you can attach it to any EC2 instance in the same Availability Zone. After you attach a volume, it appears as a native block device similar to a hard drive or other physical device. At that point, the instance can interact with the volume just as it would with a local drive.

EFS

EFS is Elastic File Storage and is basically NFS or a network drive. To use it, you have to mount it in the operating system.

Elastic Block Storage (EBS) Pricing

Amazon EBS General Purpose SSD (gp2) volumes
$0.11 per GB-month of provisioned storage

Amazon EBS Provisioned IOPS SSD (io1) volumes
$0.138 per GB-month of provisioned storage
$0.072 per provisioned IOPS-month

Amazon EBS Throughput Optimized HDD (st1) volumes
$0.05 per GB-month of provisioned storage

Amazon EBS Cold HDD (sc1) volumes
$0.028 per GB-month of provisioned storage

Amazon EBS Snapshots to Amazon S3
$0.05 per GB-month of data stored

https://aws.amazon.com/ebs/pricing

Elastic File Storage (EFS) Pricing

US East (N. Virginia)
$0.30/GB-month

US East (Ohio)
$0.30/GB-month

US West (Oregon)
$0.30/GB-month

EU (Ireland)
$0.33/GB-month

https://aws.amazon.com/efs/pricing

Virtual Private Cloud (VPC) Dashboard

https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1

Helpful YouTube Video

https://www.youtube.com/watch?v=TjVWpNZfTPE

Default Username and Password

ssh -i your-ssh-key.pem ubuntu@external.dns.name

Thanks AskUbuntu.

HOWTO: COMMAND LINE

Run the AWS CLI in Docker

#!/usr/bin/env bash
rootdir="${HOME}/.config/aws-cli-root"
if [[ ! -d "${rootdir}" ]]; then
  echo "${rootdir} does not exist"
  exit 1
fi
docker run -it -v "${rootdir}":/root/.aws amazon/aws-cli $@

Install the AWS CLI version 2 on Linux

curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
sudo ldconfig
which aws
aws --version

https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install

List SSH Keys

aws --profile awscli --region eu-west-2 ec2 describe-key-pairs

Create SSH Key

aws --profile awscli --region eu-west-2 ec2 create-key-pair --key-name <value>

Instance Create

Launch the cheapest EC2 instance in London with Ubuntu Server 20.04 using a specific SSH key and tag it with a name... 

aws --profile awscli --region eu-west-2 ec2 run-instances --image-id ami-0194c3e07668a7e36 --instance-type t2.nano --count 1 --subnet-id subnet-xxxxxxx --security-group-ids sg-xxxxxxxxxxxxx --key-name xxxxxxx --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=awscli-test-1}]' 'ResourceType=volume,Tags=[{Key=Name,Value=awscli-test-1}]'

Instance List

aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId'

or 

aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text

or 

aws ec2 describe-instances | jq '.Reservations[] | .Instances[] | {Id: .InstanceId, PublicAddress: .PublicIpAddress, PrivateAddress: .PrivateIpAddress}'

or 

aws ec2 describe-instances --filters "Name=tag:Name,Values=awscli-test-1"

or 

aws --profile default ec2 --region eu-west-2 describe-instances --query 'Reservations[].Instances[].{InstanceID:InstanceId, State:State.Name, Type:InstanceType, PrivateIpAddress:PrivateIpAddress, PublicIpAddress:PublicIpAddress}'

or 

aws --profile default ec2 --region eu-west-2 describe-instances --query 'Reservations[].Instances[].{InstanceID:InstanceId, State:State.Name, Type:InstanceType, PrivateIpAddress:PrivateIpAddress, PublicIpAddress:PublicIpAddress, Name:Tags[?Key==`Name`].Value[]}' | jq

Instance State

aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].State' --output text

Instance Tag

aws --profile awscli --region eu-west-2 ec2 create-tags --resources "i-xxxxxxxxxxxxxxx" --tags "Key=Name,Value=awscli-test-1"

Instance Stop Start

aws --region eu-central-1 ec2 stop-instances --dry-run --instance-ids i-XXXXXXXX

aws --region eu-central-1 ec2 stop-instances --instance-ids i-XXXXXXXX

aws --region eu-central-1 ec2 describe-instance-status --instance-ids i-XXXXXXXX --output json

aws --region eu-central-1 ec2 start-instances --instance-ids i-XXXXXXXX

aws --profile awscli --region eu-west-2 ec2 stop-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text`"

aws --profile awscli --region eu-west-2 ec2 start-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --filters "Name=tag:Name,Values=awscli-test-1" --output text`"

aws --profile awscli --region eu-west-2 ec2 stop-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --filters "Name=tag:Name,Values=awscli-test-1" --output text`"

aws --region eu-central-1 ec2 describe-instance-status --instance-ids i-XXXXXXXX --output text | grep 'STATE' | awk '{ print $3 }'

Instance Reboot

aws --profile awscli --region eu-west-1 ec2 reboot-instances --instance-ids i-XXXXXXXX

Get Static IP Address

From the server ... 

curl -s https://checkip.amazonaws.com

TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/public-ipv4

From your desktop ... 

aws ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].PublicIpAddress' --output text

Show IP Addresses

aws --profile awscli --region eu-west-2 ec2 describe-instances --filters "Name=tag:Name,Values=awscli-test-1" | jq '.Reservations[] | .Instances[] | {Id: .InstanceId, PublicAddress: .PublicIpAddress, PrivateAddress: .PrivateIpAddress}'

Volume Snapshots

aws ec2 create-snapshot --region eu-central-1 --volume-id vol-XXXXXXXX --description "aws cli test"

aws ec2 describe-snapshots --region eu-central-1 --owner-ids XXXXXXXXXX

aws ec2 copy-snapshot --region eu-central-1 --source-region eu-central-1 --source-snapshot-id snap-XXXXXXXXXXXXXXX --destination-region eu-west-2

Create New Volume From Snapshot

1. Create a new volume from the backup snapshot by using the following command: 

ec2-create-volume –z us-west-1b –snapshot MySnapshotName

2. Within the Amazon EC2 instance, unmount the existing volume (e.g., by using umount in Linux or the Logical Volume Manager in Windows). 3. Detach the existing volume from the instance by using the following command: 

ec2-detach-volume OldVolume

4. Attach the new volume that was created from the snapshot by using the following command: 

ec2-attach-volume VolumeID –I InstanceID –d Device

5. Remount the volume on the running instance.

Security Groups / Firewall

Add Rule

Dynamic My IP... 

aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr `curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'`/32

IMAPS... 

aws ec2 authorize-security-group-ingress --group-name imaps --protocol tcp --port 993 --cidr `curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'`/32

Or manually... 

aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr 123.456.789.10/32

TCP ports 80 and 443 from anywhere on rule group launch-wizard-3... 

aws ec2 authorize-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 80 --cidr 0.0.0.0/32
aws ec2 authorize-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 443 --cidr 0.0.0.0/32

Delete Rule

aws ec2 revoke-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr 123.456.789.10/32

aws ec2 revoke-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 80 --cidr 0.0.0.0/32
aws ec2 revoke-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 443 --cidr 0.0.0.0/32

Check Rules

aws --profile default ec2 describe-security-groups --group-names launch-wizard-3

Increase EBS Volume Size

https://docs.aws.amazon.com/ebs/latest/userguide/recognize-expanded-volume-linux.html

This is easeiest if you log in to the instance and do the commands there, because you can then Extend the file system after resizing an EBS volume.

Increase the disk size... 

sudo -i
aws ec2 modify-volume --volume-id vol-XXXXXXXXXXXXX --size <gbsize>
aws ec2 describe-volumes-modifications --volume-ids vol-XXXXXXXXXXXXX

Confirm that the volume modification succeeded and that it is in the 'completed' or 'in-use' state.

Increase the partition size... 

lsblk
growpart /dev/xvda 1
lsblk

Find out what Linux filesystem you are using... 

df -H -T

Expand the ext4 filesystem... 

resize2fs /dev/xvda1

Expand the xfs filesystem... 

xfs_growfs -d /

Reboot the instance... 

reboot

CLI Query Examples

https://how.wtf/aws-cli-query-examples.html

HOWTO: Increase Disk Size

NEW

  1. stop instance
  2. create snapshot
  3. increase volume size
  4. wait for volume state to go through 'resizing' 'optimising' and then 'complete'
  5. start instance

OLD

sudo -i
growpart /dev/xvda 1
resize2fs /dev/xvda1
reboot

FIX NO SPACE LEFT ON DEVICE

  1. ssh in to vm
  2. switch to root
  3. uname -a to find out which kernel is being used
  4. rm -rfv /boot/*kernel-version*
  5. repeat to remove all but current kernel
  6. apt-get autoclean
  7. apt-get clean

HOWTO: TAKE SCREENSHOT

How to Take a Screenshot of an Unreachable Instance

  1. Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
  2. In the left navigation pane, choose Instances.
  3. Select the instance to capture.
  4. Choose Actions, Instance Settings.
  5. Choose Get Instance Screenshot.
  6. Right-click on the image to download and save it.

Thanks - http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/screenshot-service.html#how-to-ics

HOWTO: USAGE - SCRIPT

#!/bin/bash
S3BUCKET=s3://bucket.name;
S3DU=$(s3cmd du ${S3BUCKET} |awk '{print $1}');
S3OBJECTS=$(s3cmd du ${S3BUCKET} |awk '{print $2}');
S3KB=$(echo "scale=2; ${S3DU} / 1024" | bc -l);
S3MB=$(echo "scale=2; ${S3DU} / 1024 / 1024" | bc -l);
S3GB=$(echo "scale=2; ${S3DU} / 1024 / 1024 / 1024" | bc -l);
S3COST=$(echo "scale=2; ${S3GB} * 0.02" | bc -l);
echo "${S3BUCKET}";
echo "${S3OBJECTS} Objects";
echo "${S3KB} KB";
echo "${S3MB} MB";
echo "${S3GB} GB";
echo "${S3COST} GBP";
exit 0;

HOWTO: USERS

Create User Restricted To EC2

Identity and Access Management > Create Individual Users > Manage Users > Create New Users > Friendly Name > Create > Download Credentials

Identity and Access Management > Select User > User Actions > Manage Password > Create > Download Credentials

Identity and Access Management > Policies > Policy Type > Filter > EC2 > AmazonEC2FullAccess > Select > Policy Actions > Attach > Select User > Attach Policy

HOWTO: FIX:

SSH Connection

SSH might have stop running due to an error in the config file.

To fix it...

  1. Stop the Instance
  2. Detach the EBS Volume (making a note of the root device name e.g. /dev/sda1)
  3. Attach the EBS Volume as a secondary Volume to another Instance
  4. Mount the secondary Volume
  5. Fix the errors
  6. Unmount the secondary Volume
  7. Detach the secondary Volume
  8. Reattach to your original Instance making sure you specify /dev/sda1 as the root device

Thanks - https://stackoverflow.com/questions/14026148/running-ec2-instance-suddenly-refuses-ssh-connection/21563478#21563478

Shutting Down Instance

http://aws.amazon.com/instance-help/#ebs-stuck-stopping

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesStopping.html

Locale Error

If you receive this error on SSH login... 

bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.utf8)

...then fix it with these commands... 

sudo aptitude install language-pack-en-base
sudo update-locale
LC_ALL="en_GB.UTF-8" sudo locale-gen
sudo dpkg-reconfigure locales
locale
sudo nano /etc/environment
     LANG="en_GB.UTF-8" # Add this line to the file
sudo reboot

NOTE: Even if you do not receive the error message there is no harm in undertaking the instructions above.

Thanks AskUbuntu.

HOWTO: OPENVPN SERVER

https://codesandbolts.com/openvpn-aws-ec2-2/

HOWTO: WEB SERVER

Installation

sudo apt-get install lighttpd
sudo mkdir /var/www/default
sudo touch /var/www/default/favicon.ico
sudo touch /var/www/default/index.html
sudo nano /etc/lighttpd/lighttpd.conf
  server.document-root        = "/var/www/default"
sudo service lighttpd restart

Log File Setup

sudo nano /etc/lighttpd/lighttpd.conf

Enable the access log module... 

 server.modules = (
   "mod_accesslog", # Add this line in the server.modules section
)

Create and add the access log line... 

accesslog.filename = "/var/log/lighttpd/access.log"

You can add an optional log file format line... 

accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""

Reload the configuarion... 

sudo service lighttpd reload

Thanks - http://www.error418.org/2013/02/lighttpd-access-log.html

Log File Rotation

  • NOTE In the following tutorial replace yourdomain.com with your own domain name.
sudo touch /etc/logrotate.d/lighttpd.yourdomainname.com
sudo nano /etc/logrotate.d/lighttpd.yourdomainname.com

/var/www/yourdomainname.com/logs/*.log {
   monthly
   missingok
   rotate 12
   compress
   delaycompress
   notifempty
   sharedscripts
   postrotate
        if [ -x /usr/sbin/invoke-rc.d ]; then \
           invoke-rc.d lighttpd reopen-logs > /dev/null 2>&1; \
        else \
           /etc/init.d/lighttpd reopen-logs > /dev/null 2>&1; \
        fi; \
   endscript
}

Save (CTRL+o) and exit (CTRL+x).

Thanks to Cyberciti.

Virtual Hosts

  • NOTE 1: Make sure you move the default document root (shown below)
  • NOTE 2: In the following tutorial replace yourdomain.com with your own domain name.
  • NOTE 3: The following tutorial uses ftpuser1 replace this with your choice of username, i.e. jbloggs.
  • NOTE 4: Items shown like this # This is a comment are just that comments do not run in the terminal or paste them in to files.
Do this first...

Lighttpd#HOWTO:_Move_Default_Document_Root 

sudo mkdir -p /var/www/yourdomainname.com/{html,logs}
sudo useradd -d /var/www/yourdomainname.com/ -s /usr/sbin/nologin -g www-data ftpuser1
sudo passwd ftpuser1
sudo id ftpuser1
sudo touch /var/www/yourdomainname.com/logs/{access,error}.log
sudo touch /var/www/yourdomainname.com/html/{favicon.ico,index.html}
sudo echo "yourdomain.com" > /var/www/yourdomainname.com/html/index.html
sudo chown -R ftpuser1:www-data /var/www/yourdomainname.com/
sudo chmod g+w /var/www/yourdomainname.com/logs/*
sudo echo "include yourdomainname.com.conf" >> /etc/lighttpd/lighttpd.conf
sudo nano /etc/lighttpd/yourdomainname.com.conf
$HTTP["host"] =~ "(^|\.)yourdomainname.com$" {
  url.redirect = (
    "^/(.*)" => "http://www.yourdomain.com/$1"
  )
  server.document-root = "/var/www/yourdomainname.com/html"
  server.errorlog = "/var/www/yourdomainname.com/logs/error.log"
  accesslog.filename = "/var/www/yourdomainname.com/logs/access.log"
  # uncomment below for wordpress
  # url.rewrite-if-not-file = ( "^/(wp-.+).*/?" => "$0", "^/keyword/([A-Za-z_0-9\-]+)/?$" => "/index.php?keyword=$1", "^/.*?(\?.*)?$" => "/index.php$1" )
}
sudo service lighttpd reload

Thanks to Cyberciti.

HOWTO: ADD TO SECURITY GROUPS

Allow PING

By default EC2 will add a ping rule based on the IP address you are using when you create an machine instance, so to ping from a different IP address you will need to create another rule. 

EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > / 
                Edit > Add Rule > (Type) Custom ICMP Rule > (Protocol) Echo Request > (Source) Custom IP > Save

Allow HTTP & HTTPS Ports

Separate rules are required for both protocols. 

EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > /
                Edit > Add Rule > (Type) TCP > Source (Anywhere, for the entire world) > Save

Additional SSH Locations

By default EC2 will add a SSH login rule based on the IP address you are using when you create an machine instance, so to log in via a different IP address you will need to create another rule. 

EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > / 
                Edit > Add Rule > (Type) SSH > (Source) Either "My IP" or "Custom IP" (best not to use the "Anywhere" option) > Save

HOWTO: CloudWatch Alarms

Using Alarm actions to reboot your VM

Even Faster Boot

Making EC2 boot time 8x faster

Install A GUI Desktop

https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/

HOWTO: ELASTIC IP REVERSE DNS

A reverse Domain Name System (DNS) lookup is used by email servers to track where a message originated from, and confirm that it's not spam or malicious. A reverse DNS lookup returns the domain name of an IP address. This is in contrast to a forward DNS lookup, which returns the IP address of a domain.

Configurable Reverse DNS for Amazon EC2’s Elastic IP Addresses

Sending Email from EC2

Configuring reverse DNS for an email server

Amazon Lightsail Configuring Reverse DNS For Email

HOWTO: ADD SWAP FILE

You can add a 512MB swap to your instance with these commands: 

sudo dd if=/dev/zero of=/swapfile bs=1M count=512
sudo chmod 0600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To enable it by default after reboot, add this line to /etc/fstab: 

/swapfile swap swap defaults 0 0

Thanks - http://stackoverflow.com/questions/17173972/how-do-you-add-swap-to-an-ec2-instance#17173973

HOWTO: INCREASE RAM

http://aws.amazon.com/ec2/instance-types

NEW METHOD

  1. Instances > Select Instance > Actions > Instance State > Stop
  2. Instances > Select Instance > Actions > Instance Settings > Change Instance Type > Configure > Review > Change

OLD METHOD

Unfortunately, there is no easy way to increase the RAM in a running instance. You have to create a copy of the instance, launch a new instance based on the copy but with more RAM, associate the Elastic IP address with your new instance, then stop your old instance.

  1. Instances > Select Instance > Actions > Instance State > Stop
  2. Instances > Select Instance > Actions > Image > Bundle Image (AMI) > Name > Save Image
  3. Instances > Launch Instance > My AMIs > Select > Configure > Review > Start
  4. Instances > Select NEW Instance > Note ID
  5. Elastic IPs > Select Elastic IP > Actions > Associate Address > NEW Instance ID > Associate
  6. Instances > Select NEW Instance > Actions > Connect > SSH > Check RAM
  7. Instances > Select OLD Instance > Actions > Instance State > Stop

Thanks - http://imperialwicket.com/aws-upgrade-ec2-instance-memory

HOWTO: BACKUP, COPY, MOVE, DELETE:

Backup Of EC2 Instance

https://n2ws.com/blog/how-to-guides/automate-amazon-ec2-instance-backup

http://www.techrepublic.com/blog/the-enterprise-cloud/backing-up-and-restoring-snapshots-on-amazon-ec2-machines/

Copy Instance To A Different Amazon Geographic Region

This is an example of copying an Amazon EC2 Instance from the USA to Ireland.

  1. Log in to AWS Console
  2. Select AWS Region (top right of screen).
  3. Stop Instance.
  4. Create Snapshot.
  5. Copy Snapshot (to EU West Region - Ireland).
  6. Switch AWS Region.
  7. Snapshots > right click on Snapshot and choose Create Image.
  8. AMI > right click on AMI and choose Launch.
  9. Configure Instance as needed.

Official Page - https://aws.amazon.com/blogs/aws/ec2-ami-copy-between-regions/

Nice Tutorial - https://ranjan01.wordpress.com/2014/01/03/how-to-migrate-copy-an-ec2-instance-between-amazon-aws-regions/

Official Doc - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#migrating-instance-availability-zone

Kernel Fix - http://serverfault.com/questions/447980/correct-process-for-starting-new-ec2-instance-from-an-ebs-snapshot

First Doc - http://stackoverflow.com/questions/5402013/move-amazon-ec2-amis-between-regions-via-web-interface

Ubuntu EC2 Images - http://cloud-images.ubuntu.com/locator/ec2/

HOWTO: DELETE:

AWS calls deletion termination.

  • Select the instance to be deleted
  • From the Actions menu select 'Instance Settings' then 'Change Termination Protection'
Retrieved from ‘https://wiki.indie-it.com/index.php?title=AWS_EC2&oldid=3339’