AWS EC2
INFO:
EC2 from Amazon stands for "Elastic Compute Cloud", and is a core component of their cloud based computing services collectively know as Amazon Web Services (AWS).
NOTE: In the following tutorial replace <yourdomain.com> with your own domain name.
Volumes
Volumes are hard disk drives. They can be attached to a running virtual machine (just like plugging the cables in), and then have to be partitioned, formatted and mounted.
EBS
EBS is Elastic Block Storage and basically means that Amazon use a whole bunch of drives stored over several servers and then put those together to make your disk or volume. After a volume is attached to an instance, you can use it like any other physical hard drive. EBS volumes persist independently from the running life of an EC2 instance. This is just like having a physical hard drive, unplugging it and keeping the data on it. When you create an EBS volume in an Availability Zone, it is automatically replicated within that zone to prevent data loss due to failure of any single hardware component. After you create a volume, you can attach it to any EC2 instance in the same Availability Zone. After you attach a volume, it appears as a native block device similar to a hard drive or other physical device. At that point, the instance can interact with the volume just as it would with a local drive.
EFS
EFS is Elastic File Storage and is basically NFS or a network drive. To use it, you have to mount it in the operating system.
Elastic Block Storage (EBS) Pricing
Amazon EBS General Purpose SSD (gp2) volumes $0.11 per GB-month of provisioned storage Amazon EBS Provisioned IOPS SSD (io1) volumes $0.138 per GB-month of provisioned storage $0.072 per provisioned IOPS-month Amazon EBS Throughput Optimized HDD (st1) volumes $0.05 per GB-month of provisioned storage Amazon EBS Cold HDD (sc1) volumes $0.028 per GB-month of provisioned storage Amazon EBS Snapshots to Amazon S3 $0.05 per GB-month of data stored
https://aws.amazon.com/ebs/pricing
Elastic File Storage (EFS) Pricing
US East (N. Virginia) $0.30/GB-month US East (Ohio) $0.30/GB-month US West (Oregon) $0.30/GB-month EU (Ireland) $0.33/GB-month
https://aws.amazon.com/efs/pricing
Virtual Private Cloud (VPC) Dashboard
https://eu-west-1.console.aws.amazon.com/vpc/home?region=eu-west-1
Helpful YouTube Video
https://www.youtube.com/watch?v=TjVWpNZfTPE
Default Username and Password
ssh -i your-ssh-key.pem ubuntu@external.dns.name
Thanks AskUbuntu.
HOWTO: COMMAND LINE
Run the AWS CLI in Docker
#!/usr/bin/env bash rootdir="${HOME}/.config/aws-cli-root" if [[ ! -d "${rootdir}" ]]; then echo "${rootdir} does not exist" exit 1 fi docker run -it -v "${rootdir}":/root/.aws amazon/aws-cli $@
Install the AWS CLI version 2 on Linux
cd /tmp/ curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install --update sudo ldconfig which aws aws --version
https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html#cliv2-linux-install
List SSH Keys
aws --profile awscli --region eu-west-2 ec2 describe-key-pairs
Create SSH Key
aws --profile awscli --region eu-west-2 ec2 create-key-pair --key-name <value>
Instance Create
Launch the cheapest EC2 instance in London with Ubuntu Server 20.04 using a specific SSH key and tag it with a name...
aws --profile awscli --region eu-west-2 ec2 run-instances --image-id ami-0194c3e07668a7e36 --instance-type t2.nano --count 1 --subnet-id subnet-xxxxxxx --security-group-ids sg-xxxxxxxxxxxxx --key-name xxxxxxx --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=awscli-test-1}]' 'ResourceType=volume,Tags=[{Key=Name,Value=awscli-test-1}]'
Instance List
aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId'
or
aws ec2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text
or
aws ec2 describe-instances | jq '.Reservations[] | .Instances[] | {Id: .InstanceId, PublicAddress: .PublicIpAddress, PrivateAddress: .PrivateIpAddress}'
or
aws ec2 describe-instances --filters "Name=tag:Name,Values=awscli-test-1"
or
aws --profile default ec2 --region eu-west-2 describe-instances --query 'Reservations[].Instances[].{InstanceID:InstanceId, State:State.Name, Type:InstanceType, PrivateIpAddress:PrivateIpAddress, PublicIpAddress:PublicIpAddress}'
or
aws --profile default ec2 --region eu-west-2 describe-instances --query 'Reservations[].Instances[].{InstanceID:InstanceId, State:State.Name, Type:InstanceType, PrivateIpAddress:PrivateIpAddress, PublicIpAddress:PublicIpAddress, Name:Tags[?Key==`Name`].Value[]}' | jq
or
aws ec2 describe-instances --filters 'Name=tag:Name,Values=caddy-test' --query 'Reservations[*].Instances[*].InstanceId'
Instance State
aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].State' --output text
Instance Tag
aws --profile awscli --region eu-west-2 ec2 create-tags --resources "i-xxxxxxxxxxxxxxx" --tags "Key=Name,Value=awscli-test-1"
Instance Stop Start
aws --region eu-central-1 ec2 stop-instances --dry-run --instance-ids i-XXXXXXXX aws --region eu-central-1 ec2 stop-instances --instance-ids i-XXXXXXXX aws --region eu-central-1 ec2 describe-instance-status --instance-ids i-XXXXXXXX --output json aws --region eu-central-1 ec2 start-instances --instance-ids i-XXXXXXXX aws --profile awscli --region eu-west-2 ec2 stop-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --output text`" aws --profile awscli --region eu-west-2 ec2 start-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --filters "Name=tag:Name,Values=awscli-test-1" --output text`" aws --profile awscli --region eu-west-2 ec2 stop-instances --instance-ids "`aws --profile awscli ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].InstanceId' --filters "Name=tag:Name,Values=awscli-test-1" --output text`" aws --region eu-central-1 ec2 describe-instance-status --instance-ids i-XXXXXXXX --output text | grep 'STATE' | awk '{ print $3 }'
Instance Reboot
aws --profile awscli --region eu-west-1 ec2 reboot-instances --instance-ids i-XXXXXXXX
Get Static IP Address
From the server ...
curl -s https://checkip.amazonaws.com
TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` && curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://169.254.169.254/latest/meta-data/public-ipv4
From your desktop ...
aws ec2 --region eu-west-2 describe-instances --query 'Reservations[*].Instances[*].PublicIpAddress' --output text
Show IP Addresses
aws --profile awscli --region eu-west-2 ec2 describe-instances --filters "Name=tag:Name,Values=awscli-test-1" | jq '.Reservations[] | .Instances[] | {Id: .InstanceId, PublicAddress: .PublicIpAddress, PrivateAddress: .PrivateIpAddress}'
Volume Snapshots
aws ec2 create-snapshot --region eu-central-1 --volume-id vol-XXXXXXXX --description "aws cli test" aws ec2 describe-snapshots --region eu-central-1 --owner-ids XXXXXXXXXX aws ec2 copy-snapshot --region eu-central-1 --source-region eu-central-1 --source-snapshot-id snap-XXXXXXXXXXXXXXX --destination-region eu-west-2
Create New Volume From Snapshot
1. Create a new volume from the backup snapshot by using the following command:
ec2-create-volume –z us-west-1b –snapshot MySnapshotName
2. Within the Amazon EC2 instance, unmount the existing volume (e.g., by using umount in Linux or the Logical Volume Manager in Windows). 3. Detach the existing volume from the instance by using the following command:
ec2-detach-volume OldVolume
4. Attach the new volume that was created from the snapshot by using the following command:
ec2-attach-volume VolumeID –I InstanceID –d Device
5. Remount the volume on the running instance.
Security Groups / Firewall
Add Rule
Dynamic My IP...
aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr `curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'`/32
IMAPS...
aws ec2 authorize-security-group-ingress --group-name imaps --protocol tcp --port 993 --cidr `curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'`/32
Or manually...
aws ec2 authorize-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr 123.456.789.10/32
TCP ports 80 and 443 from anywhere on rule group launch-wizard-3...
aws ec2 authorize-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 80 --cidr 0.0.0.0/32 aws ec2 authorize-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 443 --cidr 0.0.0.0/32
Delete Rule
aws ec2 revoke-security-group-ingress --group-name launch-wizard-1 --protocol tcp --port 993 --cidr 123.456.789.10/32
aws ec2 revoke-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 80 --cidr 0.0.0.0/32 aws ec2 revoke-security-group-ingress --group-name launch-wizard-3 --protocol tcp --port 443 --cidr 0.0.0.0/32
Check Rules
aws --profile default ec2 describe-security-groups --group-names launch-wizard-3
Increase EBS Volume Size
https://docs.aws.amazon.com/ebs/latest/userguide/recognize-expanded-volume-linux.html
This is easeiest if you log in to the instance and do the commands there, because you can then Extend the file system after resizing an EBS volume.
Switch to root user...
sudo -i
Find instance id of this host...
wget -q -O - http://169.254.169.254/latest/meta-data/instance-id ec2metadata --instance-id
Find volume ID and GB size of disk...
aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=`ec2metadata --instance-id` --query "Volumes[*].{ID:VolumeId,Size:Size}"
Increase the disk size...
aws ec2 modify-volume --volume-id vol-XXXXXXXXXXXXX --size <gbsize>
Check the modification..
aws ec2 describe-volumes-modifications --volume-ids vol-XXXXXXXXXXXXX
As it is processing, the state changes through modifying and optimizing.
Confirm that the volume modification succeeded and that it is in the completed or in-use state.
Check the disk to show partition name and actual size and current partition usage. For example, xvda1 and 10G and 7G...
lsblk xvda 202:0 0 10G 0 disk ├─xvda1 202:1 0 7G 0 part /
Increase the partition size...
growpart /dev/xvda 1 (<-- older disks) growpart /dev/nvme0n1 1 (<-- newer disks)
Check again...
lsblk xvda 202:0 0 10G 0 disk ├─xvda1 202:1 0 9G 0 part /
Find out what Linux filesystem you are using...
df -H -T /
Expand the ext4 filesystem...
resize2fs /dev/xvda1 (<-- older disks) resize2fs /dev/nvme0n1p1 (<-- newer disks)
Expand the xfs filesystem...
xfs_growfs -d /
Reboot the instance...
reboot
CLI Query Examples
https://how.wtf/aws-cli-query-examples.html
Launch Templates
aws ec2 modify-launch-template --launch-template-id "lt-0e4c99bf82b0d2623" --default-version "9" --region "eu-west-2"
HOWTO: Increase Disk Size
NEW
- stop instance
- create snapshot
- increase volume size
- wait for volume state to go through 'resizing' 'optimising' and then 'complete'
- start instance
OLD
sudo -i growpart /dev/xvda 1 resize2fs /dev/xvda1 reboot
FIX NO SPACE LEFT ON DEVICE
- ssh in to vm
- switch to root
- uname -a to find out which kernel is being used
- rm -rfv /boot/*kernel-version*
- repeat to remove all but current kernel
- apt-get autoclean
- apt-get clean
HOWTO: TAKE SCREENSHOT
How to Take a Screenshot of an Unreachable Instance
- Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
- In the left navigation pane, choose Instances.
- Select the instance to capture.
- Choose Actions, Instance Settings.
- Choose Get Instance Screenshot.
- Right-click on the image to download and save it.
Thanks - http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/screenshot-service.html#how-to-ics
HOWTO: USAGE - SCRIPT
#!/bin/bash S3BUCKET=s3://bucket.name; S3DU=$(s3cmd du ${S3BUCKET} |awk '{print $1}'); S3OBJECTS=$(s3cmd du ${S3BUCKET} |awk '{print $2}'); S3KB=$(echo "scale=2; ${S3DU} / 1024" | bc -l); S3MB=$(echo "scale=2; ${S3DU} / 1024 / 1024" | bc -l); S3GB=$(echo "scale=2; ${S3DU} / 1024 / 1024 / 1024" | bc -l); S3COST=$(echo "scale=2; ${S3GB} * 0.02" | bc -l); echo "${S3BUCKET}"; echo "${S3OBJECTS} Objects"; echo "${S3KB} KB"; echo "${S3MB} MB"; echo "${S3GB} GB"; echo "${S3COST} GBP"; exit 0;
HOWTO: USERS
Create User Restricted To EC2
Identity and Access Management > Create Individual Users > Manage Users > Create New Users > Friendly Name > Create > Download Credentials Identity and Access Management > Select User > User Actions > Manage Password > Create > Download Credentials Identity and Access Management > Policies > Policy Type > Filter > EC2 > AmazonEC2FullAccess > Select > Policy Actions > Attach > Select User > Attach Policy
HOWTO: FIX:
SSH Connection
SSH might have stop running due to an error in the config file.
To fix it...
- Stop the Instance
- Detach the EBS Volume (making a note of the root device name e.g. /dev/sda1)
- Attach the EBS Volume as a secondary Volume to another Instance
- Mount the secondary Volume
- Fix the errors
- Unmount the secondary Volume
- Detach the secondary Volume
- Reattach to your original Instance making sure you specify /dev/sda1 as the root device
Shutting Down Instance
http://aws.amazon.com/instance-help/#ebs-stuck-stopping
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/TroubleshootingInstancesStopping.html
Locale Error
If you receive this error on SSH login...
bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.utf8)
...then fix it with these commands...
sudo aptitude install language-pack-en-base sudo update-locale LC_ALL="en_GB.UTF-8" sudo locale-gen sudo dpkg-reconfigure locales locale sudo nano /etc/environment LANG="en_GB.UTF-8" # Add this line to the file sudo reboot
NOTE: Even if you do not receive the error message there is no harm in undertaking the instructions above.
Thanks AskUbuntu.
HOWTO: OPENVPN SERVER
https://codesandbolts.com/openvpn-aws-ec2-2/
HOWTO: WEB SERVER
Installation
sudo apt-get install lighttpd sudo mkdir /var/www/default sudo touch /var/www/default/favicon.ico sudo touch /var/www/default/index.html sudo nano /etc/lighttpd/lighttpd.conf server.document-root = "/var/www/default" sudo service lighttpd restart
Log File Setup
sudo nano /etc/lighttpd/lighttpd.conf
Enable the access log module...
server.modules = ( "mod_accesslog", # Add this line in the server.modules section )
Create and add the access log line...
accesslog.filename = "/var/log/lighttpd/access.log"
You can add an optional log file format line...
accesslog.format = "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
Reload the configuarion...
sudo service lighttpd reload
Thanks - http://www.error418.org/2013/02/lighttpd-access-log.html
Log File Rotation
- NOTE In the following tutorial replace yourdomain.com with your own domain name.
sudo touch /etc/logrotate.d/lighttpd.yourdomainname.com sudo nano /etc/logrotate.d/lighttpd.yourdomainname.com /var/www/yourdomainname.com/logs/*.log { monthly missingok rotate 12 compress delaycompress notifempty sharedscripts postrotate if [ -x /usr/sbin/invoke-rc.d ]; then \ invoke-rc.d lighttpd reopen-logs > /dev/null 2>&1; \ else \ /etc/init.d/lighttpd reopen-logs > /dev/null 2>&1; \ fi; \ endscript }
Save (CTRL+o) and exit (CTRL+x).
Thanks to Cyberciti.
Virtual Hosts
- NOTE 1: Make sure you move the default document root (shown below)
- NOTE 2: In the following tutorial replace yourdomain.com with your own domain name.
- NOTE 3: The following tutorial uses ftpuser1 replace this with your choice of username, i.e. jbloggs.
- NOTE 4: Items shown like this # This is a comment are just that comments do not run in the terminal or paste them in to files.
Do this first...
Lighttpd#HOWTO:_Move_Default_Document_Root
sudo mkdir -p /var/www/yourdomainname.com/{html,logs} sudo useradd -d /var/www/yourdomainname.com/ -s /usr/sbin/nologin -g www-data ftpuser1 sudo passwd ftpuser1 sudo id ftpuser1 sudo touch /var/www/yourdomainname.com/logs/{access,error}.log sudo touch /var/www/yourdomainname.com/html/{favicon.ico,index.html} sudo echo "yourdomain.com" > /var/www/yourdomainname.com/html/index.html sudo chown -R ftpuser1:www-data /var/www/yourdomainname.com/ sudo chmod g+w /var/www/yourdomainname.com/logs/* sudo echo "include yourdomainname.com.conf" >> /etc/lighttpd/lighttpd.conf sudo nano /etc/lighttpd/yourdomainname.com.conf $HTTP["host"] =~ "(^|\.)yourdomainname.com$" { url.redirect = ( "^/(.*)" => "http://www.yourdomain.com/$1" ) server.document-root = "/var/www/yourdomainname.com/html" server.errorlog = "/var/www/yourdomainname.com/logs/error.log" accesslog.filename = "/var/www/yourdomainname.com/logs/access.log" # uncomment below for wordpress # url.rewrite-if-not-file = ( "^/(wp-.+).*/?" => "$0", "^/keyword/([A-Za-z_0-9\-]+)/?$" => "/index.php?keyword=$1", "^/.*?(\?.*)?$" => "/index.php$1" ) } sudo service lighttpd reload
Thanks to Cyberciti.
HOWTO: ADD TO SECURITY GROUPS
Allow PING
By default EC2 will add a ping rule based on the IP address you are using when you create an machine instance, so to ping from a different IP address you will need to create another rule.
EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > / Edit > Add Rule > (Type) Custom ICMP Rule > (Protocol) Echo Request > (Source) Custom IP > Save
Allow HTTP & HTTPS Ports
Separate rules are required for both protocols.
EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > / Edit > Add Rule > (Type) TCP > Source (Anywhere, for the entire world) > Save
Additional SSH Locations
By default EC2 will add a SSH login rule based on the IP address you are using when you create an machine instance, so to log in via a different IP address you will need to create another rule.
EC2 Dashboard > Network & Security > Security Groups > Select the desired Group ID > Inbound (tab) > / Edit > Add Rule > (Type) SSH > (Source) Either "My IP" or "Custom IP" (best not to use the "Anywhere" option) > Save
HOWTO: CloudWatch Alarms
Using Alarm actions to reboot your VM
Even Faster Boot
Making EC2 boot time 8x faster
Install A GUI Desktop
https://aws.amazon.com/premiumsupport/knowledge-center/ec2-linux-2-install-gui/
HOWTO: ELASTIC IP REVERSE DNS
A reverse Domain Name System (DNS) lookup is used by email servers to track where a message originated from, and confirm that it's not spam or malicious. A reverse DNS lookup returns the domain name of an IP address. This is in contrast to a forward DNS lookup, which returns the IP address of a domain.
Configurable Reverse DNS for Amazon EC2’s Elastic IP Addresses
Configuring reverse DNS for an email server
Amazon Lightsail Configuring Reverse DNS For Email
HOWTO: ADD SWAP FILE
You can add a 512MB swap to your instance with these commands:
sudo dd if=/dev/zero of=/swapfile bs=1M count=512 sudo chmod 0600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile
To enable it by default after reboot, add this line to /etc/fstab:
/swapfile swap swap defaults 0 0
Thanks - http://stackoverflow.com/questions/17173972/how-do-you-add-swap-to-an-ec2-instance#17173973
HOWTO: INCREASE RAM OR CPU
http://aws.amazon.com/ec2/instance-types
NEW METHOD
- Instances > Select Instance > Actions > Instance State > Stop
- Instances > Select Instance > Actions > Instance Settings > Change Instance Type > Configure > Review > Change
OLD METHOD
Unfortunately, there is no easy way to increase the RAM in a running instance. You have to create a copy of the instance, launch a new instance based on the copy but with more RAM, associate the Elastic IP address with your new instance, then stop your old instance.
- Instances > Select Instance > Actions > Instance State > Stop
- Instances > Select Instance > Actions > Image > Bundle Image (AMI) > Name > Save Image
- Instances > Launch Instance > My AMIs > Select > Configure > Review > Start
- Instances > Select NEW Instance > Note ID
- Elastic IPs > Select Elastic IP > Actions > Associate Address > NEW Instance ID > Associate
- Instances > Select NEW Instance > Actions > Connect > SSH > Check RAM
- Instances > Select OLD Instance > Actions > Instance State > Stop
Thanks - http://imperialwicket.com/aws-upgrade-ec2-instance-memory
HOWTO: BACKUP, COPY, MOVE, DELETE:
Backup Of EC2 Instance
https://n2ws.com/blog/how-to-guides/automate-amazon-ec2-instance-backup
Copy Instance To A Different Amazon Geographic Region
This is an example of copying an Amazon EC2 Instance from the USA to Ireland.
- Log in to AWS Console
- Select AWS Region (top right of screen).
- Stop Instance.
- Create Snapshot.
- Copy Snapshot (to EU West Region - Ireland).
- Switch AWS Region.
- Snapshots > right click on Snapshot and choose Create Image.
- AMI > right click on AMI and choose Launch.
- Configure Instance as needed.
Official Page - https://aws.amazon.com/blogs/aws/ec2-ami-copy-between-regions/
Nice Tutorial - https://ranjan01.wordpress.com/2014/01/03/how-to-migrate-copy-an-ec2-instance-between-amazon-aws-regions/
Official Doc - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#migrating-instance-availability-zone
Kernel Fix - http://serverfault.com/questions/447980/correct-process-for-starting-new-ec2-instance-from-an-ebs-snapshot
First Doc - http://stackoverflow.com/questions/5402013/move-amazon-ec2-amis-between-regions-via-web-interface
Ubuntu EC2 Images - http://cloud-images.ubuntu.com/locator/ec2/
HOWTO: DELETE:
AWS calls deletion termination.
- Select the instance to be deleted
- From the Actions menu select 'Instance Settings' then 'Change Termination Protection'