Difference between revisions of "BASH"

From Indie IT Wiki
Line 59: Line 59:
 
== How to remove trailing whitespaces in a file ==
 
== How to remove trailing whitespaces in a file ==
  
  sed -i 's/[[:space:]]*$//' inputfile.txt
+
  <nowiki>sed -i 's/[[:space:]]*$//' inputfile.txt</nowiki>
  
 
== How to display a Specific Line in a Text File ==
 
== How to display a Specific Line in a Text File ==

Revision as of 13:42, 1 November 2021

In terminal examples anything shown in bold after a command is a comment and is not to be run, e.g:

sudo apt-get install fish Then scratch your ear

So in the above Then scratch your ear is not to be run in the terminal

BASH AWK

BASH Convert File Formats

BASH CRON

BASH curl

BASH Directories

BASH Disable

BASH Disk Tools

BASH File Manipulation

BASH Find

BASH Fix

BASH Hardware Discovery

BASH History

BASH IP Tables

BASH List

BASH Manipulate Files With Special Characters and Spaces

BASH Memory

BASH Modification

BASH MP3

BASH Networking

BASH PDF

BASH Shorewall

BASH Terminal Process

BASH Time and Date

BASH Website/Page Related

BASH Useful Tools

BASH Users

How to remove trailing whitespaces in a file

sed -i 's/[[:space:]]*$//' inputfile.txt

How to display a Specific Line in a Text File

sed -n '13p' file.txt

https://linuxhandbook.com/display-specific-lines/

How to Run a Local Shell Script on a Remote SSH Server

ssh user@remotehost 'bash -s' < script.sh

https://www.cloudsavvyit.com/14216/how-to-run-a-local-shell-script-on-a-remote-ssh-server/amp/

Create a Folder for Each Movie

cd /path/to/your/movies/files/
find . -maxdepth 1 -type f -iname "*.mkv" -exec sh -c 'mkdir "${1%.*}" ; mv "${1%}" "${1%.*}" ' _ {} \;

Strip HTML From Text

| sed 's|<[^>]*>||g'

Remove up to last backslash from URL using sed

sed 's/^.*[/]//'

e.g.

curl -I -s "https://github.com/docker/compose/releases/latest" | grep 'location:' | sed 's/^.*[/]//'

Unzip Multiple Files

The trick is to use a single quote around the zip filenames...

unzip '*.zip'
unzip 'CSV*.zip'

Delete Blank Lines

sed '/^$/d'
cat file.txt | sed '/^$/d'

Show Just Network IP Address

ip -4 addr show | grep -oP '(?<=inet\s)\d+(\.\d+){3}' | grep -v '127'

Sort Text File Containing IP Addresses

Using a dot as a delimiter, sort by the 3rd column in numerical order, then sort by the 4th column in numerical order :-)

sort -t . -k 3,3n -k 4,4n

https://www.cyberciti.biz/faq/unix-linux-shell-script-sorting-ip-addresses/

Execute Bash Script Directly From a URL

For example the FritzFrog Detection script...

sudo curl -s https://raw.githubusercontent.com/guardicore/labs_campaigns/master/FritzFrog/detect_fritzfrog.sh | bash

On an AWS EC2 or AWS Lightsail, you can even run it without being logged in to a full SSH session...

ssh my-ec2-01 sudo curl -s https://raw.githubusercontent.com/guardicore/labs_campaigns/master/FritzFrog/detect_fritzfrog.sh | bash

Redirect Output and Error to a Black Hole

command >/dev/null 2>&1

Extract Substring From Variable

variable2=${variable1:offset:length}

For example, the string '0copy' can be cut in two with...

AUDIOTRACK=${AUDIO:0:1}
AUDIOCOPY=${AUDIO:1:4}

...and tested with...

echo "${AUDIOTRACK}"
echo "${AUDIOCOPY}"

...which shows as...

0
copy

Show Free Privileged Ports

This command line will give you all the free port numbers not used in the /etc/services file...

cat /etc/services | awk '{ print $2 }' | tr -d './[:alpha:]' | sort -n | uniq | awk '$1<1024' | \grep "\S" | awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'

...and this shows you which port numbers are used...

cat /etc/services | awk '{ print $2 }' | tr -d './[:alpha:]' | sort -n | uniq | awk '$1<1024' | \grep "\S"

Using Netcat NC To Transfer Files Between Computers

On the host receiving the file...

nc -l port >file

On the client sending the file...

nc host port <file

ZIP Excluding Directories

zip -rv filename.zip path/ -x path/to/exclude/**\*
zip -rv filename.zip path/ -x path/to/exclude/**\* second/path/to/exclude/**\*

Check If Service Is Running And Restart If Not Running

#!/bin/bash

SERVICE=lighttpd

case "$(pidof ${SERVICE} | wc -w)" in

0)  echo "Restarting ${SERVICE}:"
    service lighttpd restart
    ;;
1)  echo "${SERVICE} already running"
    ;;

esac

exit;

Repeat String Echo x Times

echo $x{1..13}"Hello?"

Assign Default Variables

Turn on shell debugging output...

#!/bin/bash -x

Assign the default variable if not set...

: "${VARIABLE:=DEFAULT_VALUE}"

Show the default variable if not set...

echo ${grandslam:=Maria Sharapova}

Mount USB Drive by UUID

This is safer than using the traditional block names (/dev/sdcX) because drives can be swapped or get lost by the system. Modern Linux distributions now use this method when booting and mount your main hard disk drive by its UUID instead of the old way. The UUID is generated randomly by the formatting software.

Create directory for mount point...

sudo mkdir /mnt/usb

Find out the Universally Unique Identifier...

sudo blkid

Mount the USB drive using that UUID to that mount point...

sudo mount -U "0b4df3c4-69e0-4087-8162-whatever-usb-drive-is" /mnt/usb

Kill Process Name If Running

if pgrep rsync; then pkill rsync; fi

Geolocation Of IP Address From Firewall Log

sudo apt-get install geoip-bin
egrep 'Jul 24.*UFW.*DPT=22' syslog | awk '{ print $12 }' | sed 's/SRC=//' | sort -n | uniq | xargs -L 1 geoiplookup --verbose

My IP

curl --silent http://checkip.dyndns.com | grep -o --color=never -E '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'

PING With Date and Time

ping -i1 -c 1000000 1.1.1.1 | while read line; do echo `date` - $line; done

PING Your Nameservers

alias pingns="fping `grep 'nameserver' /etc/resolv.conf | awk '{ print $2 }' | tr '\n' ' '`"

PING Multiple Hosts At Once

sudo apt-get -y install fping
fping 208.67.222.222 google.co.uk 8.8.8.8

Thanks

ALIAS: Ping Your Router

alias pingr="ping -c3 `route -n | grep 'UG[ \t]' | head -n1 | awk '{print $2}'`"

HOWTO: Reboot On Connection Failure

echo '*/20 * * * * root /usr/bin/host api.telegram.org > /dev/null 2>1 || (/usr/bin/logger "Rebooting due to connectivity issue"; /sbin/shutdown -r now)' > /etc/cron.d/reboot-on-connection-failure

HOWTO: Open XFCE Terminal Run Command With Working Prompt

xfce4-terminal -e 'bash -c "/usr/local/bin/todo.sh -t ls; bash"' -T "todo"

Thanks - https://askubuntu.com/questions/980720/open-xfce-terminal-window-and-run-command-in-same-window/983865#983865

HOWTO: Work with Encrypted Disks on Command Line

LUKS cryptsetup LVM

Rescuing LVM+LUKS encrypted drive from LiveCD or LiveUSB

https://ubuntuforums.org/showthread.php?t=1432656

https://askubuntu.com/questions/653408/mounting-encrypted-luks-partition-from-live-cd

https://wiki.archlinux.org/index.php/Dm-crypt/Device_encryption#Manually_unlocking_a_partition_using_a_keyfile

HOWTO: Find Your DHCP Server

Command:

sudo nmap --script broadcast-dhcp-discover -e <network-interface>

Sample Output:

Starting Nmap 7.01 ( https://nmap.org ) at 2017-08-29 10:35 BST
Pre-scan script results:
| broadcast-dhcp-discover: 
|   Response 1 of 1: 
|     IP Offered: 192.168.1.143
|     DHCP Message Type: DHCPOFFER
|     Server Identifier: 192.168.1.1
|     IP Address Lease Time: 2m00s
|     Renewal Time Value: 1m00s
|     Rebinding Time Value: 1m45s
|     Subnet Mask: 255.255.255.0
|     Broadcast Address: 192.168.1.255
|     Domain Name: domain.co.uk
|     Domain Name Server: 192.168.1.6
|     Name Server: 192.168.1.6
|_    Router: 192.168.1.254

Thanks - https://superuser.com/questions/750359/check-if-a-dhcp-server-existing-in-my-network-using-bash

HOWTO: Edit A File While Viewing It

less file.txt
(press the V key)

Thanks - https://www.cyberciti.biz/faq/edit-file-when-youre-viewing-withmore-less

HOWTO: Find And Stop A Process By Name

This will find the Amazon S3 Tool s3cmd and stop it if it running... great for overnight backups that are still running the next morning and clogging up the office internet.

echo "$(pgrep s3cmd)"; kill "$(pgrep s3cmd)"

HOWTO: FIX:

No Space Left On Device

sudo df -ih

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/xvda1       1.0M  1.0M    0K  100% /

Find where it is eating all the space then prune a few days at a time. Below is an exmple of the Magento Session Cache not being cleared regularly and filling up the hard drive. In this command, we look for all files older than 120 days and delete them...

find /var/www/domain.com/html/var/session/ -type f -mtime +120 -exec rm {} \;

ERROR: Unable To Shutdown System

echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger

Thanks - http://askubuntu.com/questions/638062/shutdown-unable-to-shutdown-system-from-root-shell-init-bin-bash-ubuntu-14

HOWTO: Loop Through Directory Performing Action In Each

#!/bin/bash
for d in */; do
    sleep 1s
    echo "$d"
    filebot -script fn:artwork.tmdb "$d"
done

HOWTO: Play Audio CD

sudo apt-get install nvlc
nvlc cdda://

HOWTO: Convert Animating GIF Into Separate Images

convert animating.gif frames.png

Thanks - http://askubuntu.com/questions/101526/how-can-i-split-an-animated-gif-file-into-its-component-frames

HOWTO: Timestamp File Based On Another File

touch --reference=/path/to/file /this/other/file

HOWTO: prevent 'stdin: is not a tty' errors when using SSH

Add the following line to the top of your ~/.bashrc file...

[ -z "$PS1" ] && return

Thanks - http://www.vpshostingforum.com/howto-prevent-stdin-not-tty-errors-when-using-ssh-t4152.html

HOWTO: Restore Tar Backup Over SSH

On the target filesystem, switch to root, change to the directory you want to extract to, then ssh in to the source server and cat to tar...

su - root
cd /
ssh root@192.168.1.201 "cat /backup/wwwdata.tar.gz" | tar zxvf -

Thanks - http://www.cyberciti.biz/faq/howto-use-tar-command-through-network-over-ssh-session/

HOWTO: Extract Without First Directory

tar -xzvpf filename.tar.gz --strip-components=1 -C /path/to/extract/to/

e.g. the Magento tarball has the folder /magento/ in it, so we remove that with...

tar -xjvpf magento-1.9.0.1.tar.bz2 --strip-components=1 -C /home/website/public_html/

HOWTO: Sort Linux Password File By User ID (UID)

sudo sort -t : -k 3 -g /etc/passwd

-t = separator -k = column

HOWTO: Get Memcached Stats By Using Netcat

echo stats | nc 127.0.0.1 11211

HOWTO: Kick Users Off The System

Sometimes, you forget to log out from the system. To remove your old login, connect via SSH, find out the PID, then kill it.

The actual system login (via a keyboard attached) is always ttyX.

who -u

plittlefield   tty2         2014-04-29 15:40  old         3333
root           pts/2        2014-06-02 10:45    .         8336

kill -9 3333

HOWTO: Force Filesystem Check On Reboot

sudo touch /forcefsck
sudo reboot

HOWTO: Close A Program Remotely (and Nicely!) In Terminal

Yes, you could use pkill or killall, but there is a nicer way.

Log in to your PC via SSH, then...

Install the software...

sudo apt-get install wmctrl

List running apps...

paully@mythbuntu-server3:~$ DISPLAY=:0.0 wmctrl -l
0x01600004 -1 mythbuntu-server3 xfce4-panel
0x01800003 -1 mythbuntu-server3 Desktop
0x01600020 -1 mythbuntu-server3 xfce4-panel
0x01e00004  0 mythbuntu-server3 Terminal - paully@mythbuntu-server3: ~

Close the app...

DISPLAY=:0.0 wmctrl -c "Terminal"

http://manpages.ubuntu.com/manpages/quantal/en/man1/wmctrl.1.html

Show which programs you've been running the most in current session

hash | sort -n

How To Go To A Specific Line In LESS

Put +LINE_NUMBERg in front of the file you want to view...

less +50g /path/to/file

Securely forward TCP connections on local port 1234 to remote2 port 4321

ssh -L 1234:http://remote2.example.com:4321 username@remote.example.com

Download Multiple Files / URLs Using Wget -i

First, store all the download files or URLs in a text file as:

cat > download-file-list.txt
URL1
URL2
URL3
URL4

Next, give the download-file-list.txt as argument to wget using -i option as shown below.

wget -i download-file-list.txt

USB Scanner / Scanning On The Command Line

Example used is a Canon CanoScan N670U/N676U/LiDE20 flatbed scanner...

lsusb
lsusb -d 04a9:220d -v
sane-find-scanner
scanimage -L
scanimage --help -d plustek:libusb:006:004
scanimage -d plustek:libusb:006:004 --resolution 75 --format=tiff >/tmp/output.tiff

Thanks - http://theopoon.rinnovative.com/2012/08/22/canon-canoscan-n670u-on-ubuntu-12-04-server/

Array Looping

https://www.shell-tips.com/bash/arrays/

ARRAY=( one two three four )
for i in "${ARRAY[@]}"
do
    echo Showing $i...
    echo
done

How To Burn A Directory On To A DVD

sudo apt-get install dvd+rw-tools mkisofs cdrecord
mkisofs -r -o /tmp/mydvd.iso /path/to/folder
growisofs -Z /dev/dvd=/tmp/mydvd.iso

HOWTO: DD:

Enhanced dd with progress bars

sudo apt-get install dcfldd

Monitor The Progress Of dd with Pipe Viewer

dd if=/dev/zero of=/dev/sdX bs=8M &
echo %1
ls -l /proc/XXXXX/fd
pv -d XXXXX:1

or

dd if=/dev/mmcblk0 | pv -s 4G -peta | gzip -1 > /media/sda1/sd_backup.img.gz

This displays a progress indicator like the following.

0:15:01 [1.57MB/s] [====================>               ] 51% ETA 0:14:06

Progress Of DD Command

Begin dd in 1 terminal window. This will wipe an external USB drive...

dd if=/dev/zero bs=1M of=/dev/sdb

In another terminal window, run this...

watch -n5 'sudo kill -USR1 `pgrep ^dd`'

Now switch back to the first window...

4701814784 bytes (4.7 GB) copied, 197.487 s, 23.8 MB/s
4595+0 records in
4595+0 records out
4818206720 bytes (4.8 GB) copied, 202.506 s, 23.8 MB/s
4706+0 records in
4706+0 records out
4934598656 bytes (4.9 GB) copied, 207.578 s, 23.8 MB/s
4818+0 records in
4818+0 records out
5052039168 bytes (5.1 GB) copied, 212.662 s, 23.8 MB/s
4930+0 records in
4930+0 records out
5169479680 bytes (5.2 GB) copied, 217.737 s, 23.7 MB/s
5040+0 records in
5040+0 records out
5284823040 bytes (5.3 GB) copied, 222.771 s, 23.7 MB/s
5151+0 records in
5151+0 records out
5401214976 bytes (5.4 GB) copied, 227.784 s, 23.7 MB/s

Steganography or How To Hide A File Inside Image

How to hide files within a picture file (especially png and jpeg files) and all you'll need is a program like WinRAR or RAR.

Firstly you'll need to find two things: the first is a picture that you'll use to hide your file in, the second is the actual file you want to hide.

Put both in the same folder, open a command prompt and navigate there.

Once there you'll need to put your file that you want to hide inside a zip file (rar or zip will work just fine, put a password or whatever if you want too). After this type the following into the command prompt...

Windows

copy /b display.png + secret_message.rar fake.png

Linux

cat display.png secret_message.rar > fake.png

...where display.png is the picture in which you want to hide the file, secret_message.rar is the file you want to hide and fake.png is the file name that you want the result to be created as.

After this is done open the newly created image file in your paint package and verify that it opens correctly. To see the hidden file all you need to do is open it in your favorite compression program and your done! One hidden file that can be uploaded to image sites and so forth.

You can also put a password on the rar hidden rar file in order to add extra protection.

Change User Password One Liner Batch Mode

echo "username:password" | chpasswd

Beautify A Shell Script

http://www.arachnoid.com/linux/beautify_bash/

Upload File By FTP Command Line

ftp -u ftp://username:password@www.domain.co.uk/remote/dir/file.txt file.txt

Dictionary Attack On SSH

Oct 16 08:53:30 server2 sshd[23415]: Invalid user amcssa from 88.191.133.21
Oct 16 08:53:32 server2 sshd[23417]: Invalid user wupr from 88.191.133.21
Oct 16 08:53:34 server2 sshd[23419]: Invalid user glbt from 88.191.133.21
Oct 16 08:53:36 server2 sshd[23421]: Invalid user wusgg from 88.191.133.21
Oct 16 08:53:38 server2 sshd[23423]: Invalid user wundergr from 88.191.133.21
Oct 16 08:53:40 server2 sshd[23425]: Invalid user sos from 88.191.133.21
Oct 16 08:53:42 server2 sshd[23427]: Invalid user s0s from 88.191.133.21
Oct 16 08:53:44 server2 sshd[23429]: Invalid user sos1 from 88.191.133.21
Oct 16 08:53:46 server2 sshd[23431]: Invalid user overflow from 88.191.133.21
Oct 16 08:53:48 server2 sshd[23433]: Invalid user mariachi from 88.191.133.21
Oct 16 08:53:50 server2 sshd[23435]: Invalid user fsae from 88.191.133.21
Oct 16 08:53:52 server2 sshd[23437]: Invalid user wuma from 88.191.133.21
Oct 16 08:53:54 server2 sshd[23439]: Invalid user vsa from 88.191.133.21
Oct 16 08:53:56 server2 sshd[23441]: Invalid user pride from 88.191.133.21
Oct 16 08:53:57 server2 sshd[23443]: Invalid user yarn from 88.191.133.21
Oct 16 08:54:00 server2 sshd[23445]: Invalid user mixed from 88.191.133.21
Oct 16 08:54:02 server2 sshd[23447]: Invalid user mix3d from 88.191.133.21
Oct 16 08:54:03 server2 sshd[23449]: Invalid user washucrs from 88.191.133.21
Oct 16 08:54:05 server2 sshd[23451]: Invalid user kungfu from 88.191.133.21
Oct 16 08:54:07 server2 sshd[23453]: Invalid user lnyf from 88.191.133.21

How To Quickly Generate A Large File On The Command Line

To make a 1Mb file...

dd if=/dev/zero of=1mb_file count=1024 bs=1024

http://www.skorks.com/2010/03/how-to-quickly-generate-a-large-file-on-the-command-line-with-linux/

Foreach Loop In Script

#!/bin/bash
for i in 1 2 3
do
  /usr/bin/nmap -sP -n "192.168.0.$i" |grep 'IP'
done

Extract A Single File From A Tarball Archive

cd /tmp/
tar -xjvpf /path/to/stage4.tar.bz2 etc/conf.d/modules

Create Cool SHA Code From Date (Forum Spam Bot Prevention)

date -u +%jXfce|sha256sum|sed 's/\W//g'

Format A USB Flash Drive With Linux Filesystem

Identify your drive:

cat /proc/partitions
dmesg |tail

Repartition drive:

fdisk /dev/sdb
d
n
p
1
[enter]
[enter]
w

Format drive:

mkfs.ext3 -L "BACKUP-32GB-B" -v /dev/sdb1

Output examples:

thinkpad ~ # dmesg |tail
sd 10:0:0:0: [sdb] Assuming drive cache: write through
sd 10:0:0:0: [sdb] No Caching mode page present
sd 10:0:0:0: [sdb] Assuming drive cache: write through
   sdb: sdb1
sd 10:0:0:0: [sdb] No Caching mode page present
sd 10:0:0:0: [sdb] Assuming drive cache: write through
sd 10:0:0:0: [sdb] Attached SCSI removable disk


thinkpad ~ # cat /proc/partitions 
major minor  #blocks  name
  8        0  312571224 sda
  8        1    1228800 sda1
  8        2   83103744 sda2
  8        3  114118656 sda3
  8        4  114109695 sda4
  8       16   31696896 sdb


thinkpad ~ # fdisk /dev/sdb
Command (m for help): p
Disk /dev/sdb: 32.5 GB, 32457621504 bytes
64 heads, 32 sectors/track, 30954 cylinders, total 63393792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18

  Device Boot      Start         End      Blocks   Id  System

Command (m for help): n
Partition type:
  p   primary (0 primary, 0 extended, 4 free)
  e   extended
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-63393791, default 2048): 
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-63393791, default 63393791): 
Using default value 63393791

Command (m for help): p

Disk /dev/sdb: 32.5 GB, 32457621504 bytes
64 heads, 32 sectors/track, 30954 cylinders, total 63393792 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xc3072e18

  Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    63393791    31695872   83  Linux

Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.


thinkpad ~ # mkfs.ext3 -L "BACKUP-32GB-B" -v /dev/sdb1
mke2fs 1.42 (29-Nov-2011)
fs_types for mke2fs.conf resolution: 'ext3'
Filesystem label=BACKUP-32GB-B
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
1982464 inodes, 7923968 blocks
396198 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=4294967296
242 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000
Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done   


thinkpad ~ # umount /media/BACKUP-32GB-B/


thinkpad ~ # fsck.ext3 -v -C0 /dev/sdb1
e2fsck 1.42 (29-Nov-2011)
BACKUP-32GB-B: clean, 11/1982464 files, 168470/7923968 blocks

What Desktop Session Am I Using?

echo $DESKTOP_SESSION

Schedule With AT Command

http://www.brunolinux.com/02-The_Terminal/The_at_Command.html

at 8pm
at midnight tonight
at 3pm tomorrow

To view scheduled 'at' jobs:

atq

To delete scheduled 'at' jobs:

atrm 2

Where '2' is the job number found by using the 'atq' command

HOWTO: Use md5sum

wget http://domain.com/file.zip.md5
wget http://domain.com/file.zip
md5sum -c file.zip.md5