Difference between revisions of "BASH"
Plittlefield (talk | contribs) |
Plittlefield (talk | contribs) |
||
| Line 59: | Line 59: | ||
== How to remove trailing whitespaces in a file == | == How to remove trailing whitespaces in a file == | ||
| − | sed -i 's/[[:space:]]*$//' inputfile.txt | + | <nowiki>sed -i 's/[[:space:]]*$//' inputfile.txt</nowiki> |
== How to display a Specific Line in a Text File == | == How to display a Specific Line in a Text File == | ||
Revision as of 13:42, 1 November 2021
In terminal examples anything shown in bold after a command is a comment and is not to be run, e.g:
sudo apt-get install fish Then scratch your ear
So in the above Then scratch your ear is not to be run in the terminal
BASH Manipulate Files With Special Characters and Spaces
How to remove trailing whitespaces in a file
sed -i 's/[[:space:]]*$//' inputfile.txt
How to display a Specific Line in a Text File
sed -n '13p' file.txt
https://linuxhandbook.com/display-specific-lines/
How to Run a Local Shell Script on a Remote SSH Server
ssh user@remotehost 'bash -s' < script.sh
https://www.cloudsavvyit.com/14216/how-to-run-a-local-shell-script-on-a-remote-ssh-server/amp/
Create a Folder for Each Movie
cd /path/to/your/movies/files/
find . -maxdepth 1 -type f -iname "*.mkv" -exec sh -c 'mkdir "${1%.*}" ; mv "${1%}" "${1%.*}" ' _ {} \;
Strip HTML From Text
| sed 's|<[^>]*>||g'
Remove up to last backslash from URL using sed
sed 's/^.*[/]//'
e.g.
curl -I -s "https://github.com/docker/compose/releases/latest" | grep 'location:' | sed 's/^.*[/]//'
Unzip Multiple Files
The trick is to use a single quote around the zip filenames...
unzip '*.zip' unzip 'CSV*.zip'
Delete Blank Lines
sed '/^$/d'
cat file.txt | sed '/^$/d'
Show Just Network IP Address
ip -4 addr show | grep -oP '(?<=inet\s)\d+(\.\d+){3}' | grep -v '127'
Sort Text File Containing IP Addresses
Using a dot as a delimiter, sort by the 3rd column in numerical order, then sort by the 4th column in numerical order :-)
sort -t . -k 3,3n -k 4,4n
https://www.cyberciti.biz/faq/unix-linux-shell-script-sorting-ip-addresses/
Execute Bash Script Directly From a URL
For example the FritzFrog Detection script...
sudo curl -s https://raw.githubusercontent.com/guardicore/labs_campaigns/master/FritzFrog/detect_fritzfrog.sh | bash
On an AWS EC2 or AWS Lightsail, you can even run it without being logged in to a full SSH session...
ssh my-ec2-01 sudo curl -s https://raw.githubusercontent.com/guardicore/labs_campaigns/master/FritzFrog/detect_fritzfrog.sh | bash
Redirect Output and Error to a Black Hole
command >/dev/null 2>&1
Extract Substring From Variable
variable2=${variable1:offset:length}
For example, the string '0copy' can be cut in two with...
AUDIOTRACK=${AUDIO:0:1}
AUDIOCOPY=${AUDIO:1:4}
...and tested with...
echo "${AUDIOTRACK}"
echo "${AUDIOCOPY}"
...which shows as...
0 copy
Show Free Privileged Ports
This command line will give you all the free port numbers not used in the /etc/services file...
cat /etc/services | awk '{ print $2 }' | tr -d './[:alpha:]' | sort -n | uniq | awk '$1<1024' | \grep "\S" | awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'
...and this shows you which port numbers are used...
cat /etc/services | awk '{ print $2 }' | tr -d './[:alpha:]' | sort -n | uniq | awk '$1<1024' | \grep "\S"
Using Netcat NC To Transfer Files Between Computers
On the host receiving the file...
nc -l port >file
On the client sending the file...
nc host port <file
ZIP Excluding Directories
zip -rv filename.zip path/ -x path/to/exclude/**\*
zip -rv filename.zip path/ -x path/to/exclude/**\* second/path/to/exclude/**\*
Check If Service Is Running And Restart If Not Running
#!/bin/bash
SERVICE=lighttpd
case "$(pidof ${SERVICE} | wc -w)" in
0) echo "Restarting ${SERVICE}:"
service lighttpd restart
;;
1) echo "${SERVICE} already running"
;;
esac
exit;
Repeat String Echo x Times
echo $x{1..13}"Hello?"
Assign Default Variables
Turn on shell debugging output...
#!/bin/bash -x
Assign the default variable if not set...
: "${VARIABLE:=DEFAULT_VALUE}"
Show the default variable if not set...
echo ${grandslam:=Maria Sharapova}
Mount USB Drive by UUID
This is safer than using the traditional block names (/dev/sdcX) because drives can be swapped or get lost by the system. Modern Linux distributions now use this method when booting and mount your main hard disk drive by its UUID instead of the old way. The UUID is generated randomly by the formatting software.
Create directory for mount point...
sudo mkdir /mnt/usb
Find out the Universally Unique Identifier...
sudo blkid
Mount the USB drive using that UUID to that mount point...
sudo mount -U "0b4df3c4-69e0-4087-8162-whatever-usb-drive-is" /mnt/usb
Kill Process Name If Running
if pgrep rsync; then pkill rsync; fi
Geolocation Of IP Address From Firewall Log
sudo apt-get install geoip-bin
egrep 'Jul 24.*UFW.*DPT=22' syslog | awk '{ print $12 }' | sed 's/SRC=//' | sort -n | uniq | xargs -L 1 geoiplookup --verbose
My IP
curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'
PING With Date and Time
ping -i1 -c 1000000 1.1.1.1 | while read line; do echo `date` - $line; done
PING Your Nameservers
alias pingns="fping `grep 'nameserver' /etc/resolv.conf | awk '{ print $2 }' | tr '\n' ' '`"
PING Multiple Hosts At Once
sudo apt-get -y install fping fping 208.67.222.222 google.co.uk 8.8.8.8
ALIAS: Ping Your Router
alias pingr="ping -c3 `route -n | grep 'UG[ \t]' | head -n1 | awk '{print $2}'`"
HOWTO: Reboot On Connection Failure
echo '*/20 * * * * root /usr/bin/host api.telegram.org > /dev/null 2>1 || (/usr/bin/logger "Rebooting due to connectivity issue"; /sbin/shutdown -r now)' > /etc/cron.d/reboot-on-connection-failure
HOWTO: Open XFCE Terminal Run Command With Working Prompt
xfce4-terminal -e 'bash -c "/usr/local/bin/todo.sh -t ls; bash"' -T "todo"
HOWTO: Work with Encrypted Disks on Command Line
Rescuing LVM+LUKS encrypted drive from LiveCD or LiveUSB
https://ubuntuforums.org/showthread.php?t=1432656
https://askubuntu.com/questions/653408/mounting-encrypted-luks-partition-from-live-cd
HOWTO: Find Your DHCP Server
Command:
sudo nmap --script broadcast-dhcp-discover -e <network-interface>
Sample Output:
Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-29 10:35 BST Pre-scan script results: | broadcast-dhcp-discover: | Response 1 of 1: | IP Offered: 192.168.1.143 | DHCP Message Type: DHCPOFFER | Server Identifier: 192.168.1.1 | IP Address Lease Time: 2m00s | Renewal Time Value: 1m00s | Rebinding Time Value: 1m45s | Subnet Mask: 255.255.255.0 | Broadcast Address: 192.168.1.255 | Domain Name: domain.co.uk | Domain Name Server: 192.168.1.6 | Name Server: 192.168.1.6 |_ Router: 192.168.1.254
Thanks - https://superuser.com/questions/750359/check-if-a-dhcp-server-existing-in-my-network-using-bash
HOWTO: Edit A File While Viewing It
less file.txt (press the V key)
Thanks - https://www.cyberciti.biz/faq/edit-file-when-youre-viewing-withmore-less
HOWTO: Find And Stop A Process By Name
This will find the Amazon S3 Tool s3cmd and stop it if it running... great for overnight backups that are still running the next morning and clogging up the office internet.
echo "$(pgrep s3cmd)"; kill "$(pgrep s3cmd)"
HOWTO: FIX:
No Space Left On Device
sudo df -ih Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvda1 1.0M 1.0M 0K 100% /
Find where it is eating all the space then prune a few days at a time. Below is an exmple of the Magento Session Cache not being cleared regularly and filling up the hard drive. In this command, we look for all files older than 120 days and delete them...
find /var/www/domain.com/html/var/session/ -type f -mtime +120 -exec rm {} \;
ERROR: Unable To Shutdown System
echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger
HOWTO: Loop Through Directory Performing Action In Each
#!/bin/bash
for d in */; do
sleep 1s
echo "$d"
filebot -script fn:artwork.tmdb "$d"
done
HOWTO: Play Audio CD
sudo apt-get install nvlc nvlc cdda://
HOWTO: Convert Animating GIF Into Separate Images
convert animating.gif frames.png
Thanks - http://askubuntu.com/questions/101526/how-can-i-split-an-animated-gif-file-into-its-component-frames
HOWTO: Timestamp File Based On Another File
touch --reference=/path/to/file /this/other/file
HOWTO: prevent 'stdin: is not a tty' errors when using SSH
Add the following line to the top of your ~/.bashrc file...
[ -z "$PS1" ] && return
Thanks - http://www.vpshostingforum.com/howto-prevent-stdin-not-tty-errors-when-using-ssh-t4152.html
HOWTO: Restore Tar Backup Over SSH
On the target filesystem, switch to root, change to the directory you want to extract to, then ssh in to the source server and cat to tar...
su - root cd / ssh root@192.168.1.201 "cat /backup/wwwdata.tar.gz" | tar zxvf -
Thanks - http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/
HOWTO: Extract Without First Directory
tar -xzvpf filename.tar.gz --strip-components=1 -C /path/to/extract/to/
e.g. the Magento tarball has the folder /magento/ in it, so we remove that with...
tar -xjvpf magento-1.9.0.1.tar.bz2 --strip-components=1 -C /home/website/public_html/
HOWTO: Sort Linux Password File By User ID (UID)
sudo sort -t : -k 3 -g /etc/passwd
-t = separator -k = column
HOWTO: Get Memcached Stats By Using Netcat
echo stats | nc 127.0.0.1 11211
HOWTO: Kick Users Off The System
Sometimes, you forget to log out from the system. To remove your old login, connect via SSH, find out the PID, then kill it.
The actual system login (via a keyboard attached) is always ttyX.
who -u plittlefield tty2 2014-04-29 15:40 old 3333 root pts/2 2014-06-02 10:45 . 8336 kill -9 3333
HOWTO: Force Filesystem Check On Reboot
sudo touch /forcefsck sudo reboot
HOWTO: Close A Program Remotely (and Nicely!) In Terminal
Yes, you could use pkill or killall, but there is a nicer way.
Log in to your PC via SSH, then...
Install the software...
sudo apt-get install wmctrl
List running apps...
paully@mythbuntu-server3:~$ DISPLAY=:0.0 wmctrl -l 0x01600004 -1 mythbuntu-server3 xfce4-panel 0x01800003 -1 mythbuntu-server3 Desktop 0x01600020 -1 mythbuntu-server3 xfce4-panel 0x01e00004 0 mythbuntu-server3 Terminal - paully@mythbuntu-server3: ~
Close the app...
DISPLAY=:0.0 wmctrl -c "Terminal"
http://manpages.ubuntu.com/manpages/quantal/en/man1/wmctrl.1.html
Show which programs you've been running the most in current session
hash | sort -n
How To Go To A Specific Line In LESS
Put +LINE_NUMBERg in front of the file you want to view...
less +50g /path/to/file
Securely forward TCP connections on local port 1234 to remote2 port 4321
ssh -L 1234:http://remote2.example.com:4321 username@remote.example.com
Download Multiple Files / URLs Using Wget -i
First, store all the download files or URLs in a text file as:
cat > download-file-list.txt URL1 URL2 URL3 URL4
Next, give the download-file-list.txt as argument to wget using -i option as shown below.
wget -i download-file-list.txt
USB Scanner / Scanning On The Command Line
Example used is a Canon CanoScan N670U/N676U/LiDE20 flatbed scanner...
lsusb lsusb -d 04a9:220d -v sane-find-scanner scanimage -L scanimage --help -d plustek:libusb:006:004 scanimage -d plustek:libusb:006:004 --resolution 75 --format=tiff >/tmp/output.tiff
Thanks - http://theopoon.rinnovative.com/2012/08/22/canon-canoscan-n670u-on-ubuntu-12-04-server/
Array Looping
https://www.shell-tips.com/bash/arrays/
ARRAY=( one two three four )
for i in "${ARRAY[@]}"
do
echo Showing $i...
echo
done
How To Burn A Directory On To A DVD
sudo apt-get install dvd+rw-tools mkisofs cdrecord mkisofs -r -o /tmp/mydvd.iso /path/to/folder growisofs -Z /dev/dvd=/tmp/mydvd.iso
HOWTO: DD:
Enhanced dd with progress bars
sudo apt-get install dcfldd
Monitor The Progress Of dd with Pipe Viewer
dd if=/dev/zero of=/dev/sdX bs=8M & echo %1 ls -l /proc/XXXXX/fd pv -d XXXXX:1
or
dd if=/dev/mmcblk0 | pv -s 4G -peta | gzip -1 > /media/sda1/sd_backup.img.gz
This displays a progress indicator like the following.
0:15:01 [1.57MB/s] [====================> ] 51% ETA 0:14:06
Progress Of DD Command
Begin dd in 1 terminal window. This will wipe an external USB drive...
dd if=/dev/zero bs=1M of=/dev/sdb
In another terminal window, run this...
watch -n5 'sudo kill -USR1 `pgrep ^dd`'
Now switch back to the first window...
4701814784 bytes (4.7 GB) copied, 197.487 s, 23.8 MB/s 4595+0 records in 4595+0 records out 4818206720 bytes (4.8 GB) copied, 202.506 s, 23.8 MB/s 4706+0 records in 4706+0 records out 4934598656 bytes (4.9 GB) copied, 207.578 s, 23.8 MB/s 4818+0 records in 4818+0 records out 5052039168 bytes (5.1 GB) copied, 212.662 s, 23.8 MB/s 4930+0 records in 4930+0 records out 5169479680 bytes (5.2 GB) copied, 217.737 s, 23.7 MB/s 5040+0 records in 5040+0 records out 5284823040 bytes (5.3 GB) copied, 222.771 s, 23.7 MB/s 5151+0 records in 5151+0 records out 5401214976 bytes (5.4 GB) copied, 227.784 s, 23.7 MB/s
Steganography or How To Hide A File Inside Image
How to hide files within a picture file (especially png and jpeg files) and all you'll need is a program like WinRAR or RAR.
Firstly you'll need to find two things: the first is a picture that you'll use to hide your file in, the second is the actual file you want to hide.
Put both in the same folder, open a command prompt and navigate there.
Once there you'll need to put your file that you want to hide inside a zip file (rar or zip will work just fine, put a password or whatever if you want too). After this type the following into the command prompt...
Windows
copy /b display.png + secret_message.rar fake.png
Linux
cat display.png secret_message.rar > fake.png
...where display.png is the picture in which you want to hide the file, secret_message.rar is the file you want to hide and fake.png is the file name that you want the result to be created as.
After this is done open the newly created image file in your paint package and verify that it opens correctly. To see the hidden file all you need to do is open it in your favorite compression program and your done! One hidden file that can be uploaded to image sites and so forth.
You can also put a password on the rar hidden rar file in order to add extra protection.
Change User Password One Liner Batch Mode
echo "username:password" | chpasswd
Beautify A Shell Script
http://www.arachnoid.com/linux/beautify_bash/
Upload File By FTP Command Line
ftp -u ftp://username:password@www.domain.co.uk/remote/dir/file.txt file.txt
Dictionary Attack On SSH
Oct 16 08:53:30 server2 sshd[23415]: Invalid user amcssa from 88.191.133.21 Oct 16 08:53:32 server2 sshd[23417]: Invalid user wupr from 88.191.133.21 Oct 16 08:53:34 server2 sshd[23419]: Invalid user glbt from 88.191.133.21 Oct 16 08:53:36 server2 sshd[23421]: Invalid user wusgg from 88.191.133.21 Oct 16 08:53:38 server2 sshd[23423]: Invalid user wundergr from 88.191.133.21 Oct 16 08:53:40 server2 sshd[23425]: Invalid user sos from 88.191.133.21 Oct 16 08:53:42 server2 sshd[23427]: Invalid user s0s from 88.191.133.21 Oct 16 08:53:44 server2 sshd[23429]: Invalid user sos1 from 88.191.133.21 Oct 16 08:53:46 server2 sshd[23431]: Invalid user overflow from 88.191.133.21 Oct 16 08:53:48 server2 sshd[23433]: Invalid user mariachi from 88.191.133.21 Oct 16 08:53:50 server2 sshd[23435]: Invalid user fsae from 88.191.133.21 Oct 16 08:53:52 server2 sshd[23437]: Invalid user wuma from 88.191.133.21 Oct 16 08:53:54 server2 sshd[23439]: Invalid user vsa from 88.191.133.21 Oct 16 08:53:56 server2 sshd[23441]: Invalid user pride from 88.191.133.21 Oct 16 08:53:57 server2 sshd[23443]: Invalid user yarn from 88.191.133.21 Oct 16 08:54:00 server2 sshd[23445]: Invalid user mixed from 88.191.133.21 Oct 16 08:54:02 server2 sshd[23447]: Invalid user mix3d from 88.191.133.21 Oct 16 08:54:03 server2 sshd[23449]: Invalid user washucrs from 88.191.133.21 Oct 16 08:54:05 server2 sshd[23451]: Invalid user kungfu from 88.191.133.21 Oct 16 08:54:07 server2 sshd[23453]: Invalid user lnyf from 88.191.133.21
How To Quickly Generate A Large File On The Command Line
To make a 1Mb file...
dd if=/dev/zero of=1mb_file count=1024 bs=1024
http://www.skorks.com/2010/03/how-to-quickly-generate-a-large-file-on-the-command-line-with-linux/
Foreach Loop In Script
#!/bin/bash for i in 1 2 3 do /usr/bin/nmap -sP -n "192.168.0.$i" |grep 'IP' done
Extract A Single File From A Tarball Archive
cd /tmp/ tar -xjvpf /path/to/stage4.tar.bz2 etc/conf.d/modules
Create Cool SHA Code From Date (Forum Spam Bot Prevention)
date -u +%jXfce|sha256sum|sed 's/\W//g'
Format A USB Flash Drive With Linux Filesystem
Identify your drive:
cat /proc/partitions dmesg |tail
Repartition drive:
fdisk /dev/sdb d n p 1 [enter] [enter] w
Format drive:
mkfs.ext3 -L "BACKUP-32GB-B" -v /dev/sdb1
Output examples:
thinkpad ~ # dmesg |tail
sd 10:0:0:0: [sdb] Assuming drive cache: write through
sd 10:0:0:0: [sdb] No Caching mode page present
sd 10:0:0:0: [sdb] Assuming drive cache: write through
sdb: sdb1
sd 10:0:0:0: [sdb] No Caching mode page present
sd 10:0:0:0: [sdb] Assuming drive cache: write through
sd 10:0:0:0: [sdb] Attached SCSI removable disk
thinkpad ~ # cat /proc/partitions
major minor #blocks name
8 0 312571224 sda
8 1 1228800 sda1
8 2 83103744 sda2
8 3 114118656 sda3
8 4 114109695 sda4
8 16 31696896 sdb
thinkpad ~ # fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 32.5 GB, 32457621504 bytes
64 heads, 32 sectors/track, 30954 cylinders, total 63393792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18
Device Boot Start End Blocks Id System
Command (m for help): n
Partition type:
p primary (0 primary, 0 extended, 4 free)
e extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-63393791, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-63393791, default 63393791):
Using default value 63393791
Command (m for help): p
Disk /dev/sdb: 32.5 GB, 32457621504 bytes
64 heads, 32 sectors/track, 30954 cylinders, total 63393792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18
Device Boot Start End Blocks Id System
/dev/sdb1 2048 63393791 31695872 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
thinkpad ~ # mkfs.ext3 -L "BACKUP-32GB-B" -v /dev/sdb1
mke2fs 1.42 (29-Nov-2011)
fs_types for mke2fs.conf resolution: 'ext3'
Filesystem label=BACKUP-32GB-B
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1982464 inodes, 7923968 blocks
396198 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
242 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
thinkpad ~ # umount /media/BACKUP-32GB-B/
thinkpad ~ # fsck.ext3 -v -C0 /dev/sdb1
e2fsck 1.42 (29-Nov-2011)
BACKUP-32GB-B: clean, 11/1982464 files, 168470/7923968 blocks
What Desktop Session Am I Using?
echo $DESKTOP_SESSION
Schedule With AT Command
http://www.brunolinux.com/02-The_Terminal/The_at_Command.html
at 8pm
at midnight tonight
at 3pm tomorrow
To view scheduled 'at' jobs:
atq
To delete scheduled 'at' jobs:
atrm 2
Where '2' is the job number found by using the 'atq' command
HOWTO: Use md5sum
wget http://domain.com/file.zip.md5 wget http://domain.com/file.zip md5sum -c file.zip.md5
