CLI and Scripts Basics

Introduction
Basic Commands
File Transfer
Encryption
- Encrypt files with Gocryptfs
- Encrypt files with Veracrypt
Network-related Commands
Basic Scripts
Linux
- Install Terraform
MAC
Windows
References

Introduction

We present here a quick guide on different command-line interface (CLI) commands as well as some basic scripts.

The main goal of this guide is to demonstrate that having some core understanding of CLI and scripts can drastically increase efficiency and speed when it comes to deploying and managing workloads on the TFGrid.

Basic Commands

Update and upgrade packages

The command update ensures that you have access to the latest versions of packages available.

sudo apt update

The command upgrade downloads and installs the updates for each outdated package and dependency on your system.

sudo apt upgrade

Install Go

Here are the steps to install Go.

Install go
- ```
sudo apt install golang-go
```
Verify that go is properly installed
- ```
go version
```

Install Brew

Follow those steps to install Brew

Installation command from Brew:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Add the path to the .profile directory. Replace <user_name> by your username.

echo 'eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"' >> /home/<user_name>/.profile

Evaluation the following:

eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"

Verify the installation
- ```
brew doctor
```

Brew basic commands

To update brew in general:
- ```
brew update
```
To update a specific package:
- ```
brew update <package_name>
```
To install a package:
- ```
brew install <package_name>
```
To uninstall a package:
- ```
brew uninstall <package_name>
```
To search a package:
- ```
brew search <package_name>
```

Uninstall Brew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/uninstall.sh)"

Install Terraform with Brew

Installing Terraform with Brew is very simple by following the Terraform documentation.

Compile HashiCorp software on Homebrew's infrastructure
- ```
brew tap hashicorp/tap
```
Install Terraform
- ```
brew install hashicorp/tap/terraform
```

Yarn basic commands

Add a package
- ```
yarn add
```
Initialize the development of a package
- ```
yarn init
```
Install all the dependencies in the package.json file
- ```
yarn install
```
Publish a package to a package manager
- ```
yarn publish
```
Remove unused package from the current package
- ```
yarn remove
```
Clean the cache
- ```
yarn cache clean
```

Set default terminal

update-alternatives --config x-terminal-emulator

See the current path

pwd

List hidden files

ls -ld .?*

Display the content of a directory

You can use tree to display the files and organization of a directory:

General command
- ```
tree
```
View hidden files
- ```
tree -a
```

Vim modes and basic commands

Vim is a free and open-source, screen-based text editor program.

With Vim, you can use two modes.

Insert mode - normal text editor
- Press i
Command mode - commands to the editor
- Press ESC

Here are some basic commands:

Delete characters
- x
Undo last command
- u
Undo the whole line
- U
Go to the end of line
- A
Save and exit
- :wq
Discard all changes
- :q!
Move cursor to the start of the line
- 0
Delete the current word
- dw
Delete the current line
- dd

Check the listening ports using netstat

Use the command:

netstat

See the disk usage of different folders

du -sh *

Verify the application version

which <application_name>

Find the path of a file with only the file name

On MAC and Linux, you can use coreutils and realpath from Brew:

```
brew install coreutils
```
```
realpath file_name
```

Become the superuser (su) on Linux

You can use either command:

Option 1
- ```
sudo -i
```
Option 2
- ```
sudo -s
```

Exit a session

You can use either command depending on your shell:

```
exit
```
```
logout
```

Know the current user

You can use the following command:

```
whoami
```

See the path of a package

To see the path of a package, you can use the following command:

```
whereis <package_name>
```

Set the path of a package

export PATH=$PATH:/snap/bin

See the current path

pwd

Find the current shell

Compact version
- ```
echo $SHELL
```
Detailed version
- ```
ls -l /proc/$$/exe
```

SSH into Remote Server

Create SSH key pair
- ```
ssh-keygen
```
Install openssh-client on the local computer*
- ```
sudo apt install openssh-client
```
Install openssh-server on the remote computer*
- ```
sudo apt install openssh-server
```
Copy public key
- ```
cat ~/.ssh/id_rsa.pub
```
Create the ssh directory on the remote computer
- ```
mkdir ~/.ssh
```
Add public key in the file authorized_keys on the remote computer
- ```
nano ~/.ssh/authorized_keys
```
Check openssh-server status
- ```
sudo service ssh status
```

SSH into the remote machine

ssh <username>@<remote_server_IP_or_hostname>

*Note: For MAC, you can install openssh-server and openssh-client with Brew: brew install openssh-server and brew install openssh-client.

To enable remote login on a MAC, read this section.

Replace a string by another string in a text file

Replace one string by another (e.g. old_string, new_string)

sed -i 's/old_string/new_string/g' <file_path>/<file_name>

Use environment variables (double quotes)

sed -i "s/old_string/$env_variable/g" <file_path>/<file_name>

Replace extensions of files in a folder

Replace ext1 and ext2 by the extensions in question.

find ./ -depth -name "*.ext1" -exec sh -c 'mv "$1" "${1%.ext1}.ext2"' _ {} \;

Remove extension of files in a folder

Replace ext with the extension in question.

for file in *.ext; do
    mv -- "$file" "${file%%.ext}"
done

See the current date and time on Linux

date

Special variables in Bash Shell

Special Variables	Descriptions
$0	Name of the bash script
$1, $2...$n	Bash script arguments
$$	Process id of the current shell
$*	String containing every command-line argument
$#	Total number of arguments passed to the script
$@	Value of all the arguments passed to the script
$?	Exit status of the last executed command
$!	Process id of the last executed command
$-	Print current set of option in current shell

Gather DNS information of a website

You can use Dig to gather DNS information of a website

Template
- ```
dig <website.tld>
```
Example
- ```
dig threefold.io
```

You can also use online tools such as DNS Checker.

Partition and mount a disk

We present one of many ways to partition and mount a disk.

Create partition with gparted
- ```
sudo gparted
```
Find the disk you want to mount (e.g. sdb)
- ```
sudo fdisk -l
```
Create a directory to mount the disk to
- ```
sudo mkdir /mnt/disk
```
Open fstab
- ```
sudo nano /etc/fstab
```
Append the following to the fstab with the proper disk path (e.g. /dev/sdb) and mount point (e.g. /mnt/disk)
- ```
/dev/sdb    /mnt/disk    ext4    defaults    0    0
```
Mount the disk
- ```
sudo mount /mnt/disk
```
Add permissions (as needed)
- ```
sudo chmod -R 0777 /mnt/disk
```

File Transfer

Transfer files between local and remote computers (IPv4 and IPv6) with scp

From local to remote, write the following on the local terminal:

scp <path_to_local_file>/<filename> <remote_username>@<remote_IPv4_address>:/<remote_username>/<path_to_remote_file>/<filename>

From remote to local, you can write the following on the local terminal (more secure):

scp <remote_username>@<remote_IPv4_address>:/<remote_username>/<path_to_remote_file>/<filename> <path_to_local_file>/<file>

From remote to local, you can also write the following on the remote terminal:

scp <path_to_remote_file>/<file> <local_user>@<local_IPv4_address>:/<local_username>/<path_to_local_file>/<filename>

For IPv6, simply add -6 after scp and add \[ before and \] after the IPv6 address.

Transfer files between computers (local or remote ) with rsync

rsync is a utility for efficiently transferring and synchronizing files between a computer and a storage drive and across networked computers by comparing the modification times and sizes of files.

We show here how to transfer files between two computers. Note that at least one of the two computers must be local. This will transfer the content of the source directory into the destination directory.

From local to remote

rsync -avz --progress --delete /path/to/local/directory/ remote_user@<remote_host_or_ip>:/path/to/remote/directory

From remote to local

rsync -avz --progress --delete remote_user@<remote_host_or_ip>:/path/to/remote/directory/ /path/to/local/directory

Here is short description of the parameters used:

-a: archive mode, preserving the attributes of the files and directories
-v: verbose mode, displaying the progress of the transfer
-z: compress mode, compressing the data before transferring
--progress tells rsync to print information showing the progress of the transfer
--delete tells rsync to delete files that aren't on the sending side

Adjust reorganization of files and folders before running rsync

rsync-sidekick propagates changes from source directory to destination directory. You can run rsync-sidekick before running rsync. Make sure that Go is installed.

Install rsync-sidekick

sudo go install github.com/m-manu/rsync-sidekick@latest

Reorganize the files and folders with rsync-sidekick

rsync-sidekick /path/to/local/directory/ username@IP_Address:/path/to/remote/directory

Transfer and update files and folders with rsync

sudo rsync -avz --progress --delete --log-file=/path/to/local/directory/rsync_storage.log /path/to/local/directory/ username@IP_Address:/path/to/remote/directory

Automate backup with rsync

We show how to automate file transfers between two computers using rsync.

Create the script file
- ```
nano rsync_backup.sh
```

Write the following script with the proper paths. Here the log is saved in the same directory.

# filename: rsync_backup.sh
#!/bin/bash

sudo rsync -avz --progress --delete --log-file=/path/to/local/directory/rsync_storage.log /path/to/local/directory/ username@IP_Address:/path/to/remote/directory

Give permission

sudo chmod +x /path/to/script/rsync_backup.sh

Set a cron job to run the script periodically
- Copy your .sh file to /root:
```
sudo cp path/to/script/rsync_backup.sh /root
```
Open the cron file
- ```
sudo crontab -e
```
Add the following to run the script everyday. For this example, we set the time at 18:00PM
- ```
0 18 * * * /root/rsync_backup.sh
```

Parameters --checksum and --ignore-times with rsync

Depending on your situation, the parameters --checksum or --ignore-times can be quite useful. Note that adding either parameter will slow the transfer.

With --ignore time, you ignore both the time and size of each file. This means that you transfer all files from source to destination.
- ```
rsync --ignore-time source_folder/ destination_folder
```
With --checksum, you verify with a checksum that the files from source and destination are the same. This means that you transfer all files that have a different checksum compared source to destination.
- ```
rsync --checksum source_folder/ destination_folder
```

Trailing slashes with rsync

rsync does not act the same whether you use or not a slash ("/") at the end of the source path.

Copy content of source_folder into destination_folder to obtain the result: destination_folder/source_folder_content
- ```
rsync source_folder/ destination_folder
```
Copy source_folder into destination_folder to obtain the result: destination_folder/source_folder/source_folder_content
- ```
rsync source_folder destination_folder
```

Encryption

Encrypt files with Gocryptfs

You can use gocryptfs to encrypt files.

Install gocryptfs
- ```
apt install gocryptfs
```
Create a vault directory (e.g. vaultdir) and a mount directory (e.g. mountdir)
- ```
mkdir vaultdir mountdir
```
Initiate the vault
- ```
gocryptfs -init vaultdir
```
Mount the mount directory with the vault
- ```
gocryptfs vaultdir mountdir
```
You can now create files in the folder. For example:
- ```
touch mountdir/test.txt
```
The new file test.txt is now encrypted in the vault
- ```
ls vaultdir
```
To unmount the mountedvault folder:
- Option 1
  - ```
  fusermount -u mountdir
```
- Option 2
  - ```
  rmdir mountdir
```

Encrypt files with Veracrypt

To encrypt files, you can use Veracrypt. Let's see how to download and install Veracrypt.

Veracrypt GUI

Download the package

wget https://launchpad.net/veracrypt/trunk/1.25.9/+download/veracrypt-1.25.9-Ubuntu-22.04-amd64.deb

Install the package

dpkg -i ./veracrypt-1.25.9-Ubuntu-22.04-amd64.deb

Veracrypt console only

Download the package

wget https://launchpad.net/veracrypt/trunk/1.25.9/+download/veracrypt-console-1.25.9-Ubuntu-22.04-amd64.deb

Install the package

dpkg -i ./veracrypt-console-1.25.9-Ubuntu-22.04-amd64.deb

You can visit Veracrypt download page to get the newest releases.

To run Veracrypt
- ```
veracrypt
```
Veracrypt documentation is very complete. To begin using the application, visit the Beginner's Tutorial.

See the network connections and ports

ifconfig

See identity and info of IP address

See abuses related to an IP address:

https://www.abuseipdb.com/check/<IP_Address>

See general information of an IP address:

https://www.whois.com/whois/<IP_Address>

ip basic commands

Manage and display the state of all network
- ```
ip link
```
Display IP Addresses and property information (abbreviation of address)
- ```
ip addr
```
Display and alter the routing table
- ```
ip route
```
Manage and display multicast IP addresses
- ```
ip maddr
```
Show neighbour object
- ```
ip neigh
```
Display a list of commands and arguments for each subcommand
- ```
ip help
```
Add an address
- Template
  - ```
  ip addr add
```
- Example: set IP address to device enp0
  - ```
  ip addr add 192.168.3.4/24 dev enp0
```
Delete an address
- Template
  - ```
  ip addr del
```
- Example: set IP address to device enp0
  - ```
  ip addr del 192.168.3.4/24 dev enp0
```
Alter the status of an interface
- Template
  - ```
  ip link set
```
- Example 1: Bring interface online (here device em2)
  - ```
  ip link set em2 up
```
- Example 2: Bring interface offline (here device em2)
  - ```
  ip link set em2 down
```
Add a multicast address
- Template
  - ```
  ip maddr add
```
- Example : set IP address to device em2
  - ```
  ip maddr add 33:32:00:00:00:01 dev em2
```
Delete a multicast address
- Template
  - ```
  ip maddr del
```
- Example: set IP address to device em2
  - ```
  ip maddr del 33:32:00:00:00:01 dev em2
```
Add a routing table entry
- Template
  - ```
  ip route add
```
- Example 1: Add a default route (for all addresses) via a local gateway
  - ```
  ip route add default via 192.168.1.1 dev em1
```
- Example 2: Add a route to 192.168.3.0/24 via the gateway at 192.168.3.2
  - ```
  ip route add 192.168.3.0/24 via 192.168.3.2
```
- Example 3: Add a route to 192.168.1.0/24 that can be reached on device em1
  - ```
  ip route add 192.168.1.0/24 dev em1
```
Delete a routing table entry
- Template
  - ```
  ip route delete
```
- Example: Delete the route for 192.168.1.0/24 via the gateway at 192.168.1.1
  - ```
  ip route delete 192.168.1.0/24 via 192.168.1.1
```
Replace, or add, a route
- Template
  - ```
  ip route replace
```
- Example: Replace the defined route for 192.168.1.0/24 to use device em1
  - ```
  ip route replace 192.168.1.0/24 dev em1
```
Display the route an address will take
- Template
  - ```
  ip route get
```
- Example: Display the route taken for IP 192.168.18.25
  - ```
  ip route replace 192.168.18.25/24 dev enp0
```

References: https://www.commandlinux.com/man-page/man8/ip.8.html

Display socket statistics

Show all sockets
- ```
ss -a
```
Show detailed socket information
- ```
ss -e
```
Show timer information
- ```
ss -o
```
Do not resolve address
- ```
ss -n
```
Show process using the socket
- ```
ss -p
```

Note: You can combine parameters, e.g. ss -aeo.

References: https://www.commandlinux.com/man-page/man8/ss.8.html

Query or control network driver and hardware settings

Display ring buffer for a device (e.g. eth0)
- ```
ethtool -g eth0
```
Display driver information for a device (e.g. eth0)
- ```
ethtool -i eth0
```
Identify eth0 by sight, e.g. by causing LEDs to blink on the network port
- ```
ethtool -p eth0
```
Display network and driver statistics for a device (e.g. eth0)
- ```
ethtool -S eth0
```

References: https://man.archlinux.org/man/ethtool.8.en

See if ethernet port is active

Replace <ethernet_device> with the proper device:

cat /sys/class/net/<ethernet_device>/carrier

Add IP address to hardware port (ethernet)

Find ethernet port ID on both computers
- ```
ip a
```

Add IP address (DHCO or static)

Computer 1

ip addr add <Private_IP_Address_1>/24 dev <ethernet_interface_1>

Computer 2

ip addr add <Private_IP_Address_2>/24 dev <ethernet_interface_2>

Ping the address to confirm connection
- ```
ping -c 5 <Private_IP_Address>
```

To set and view the address for either DHCP or static, go to Networks then Details.

Private IP address range

The private IP range is the following:

10.0.0.0–10.255.255.255
172.16.0.0–172.31.255.255
192.168.0.0–192.168.255.255

Set IP Address manually

You can use the following template when you set an IP address manually:

Address
- <Private_IP_Address>
Netmask
- 255.255.255.0
Gateway
- optional

Basic Scripts

Run a script with arguments

You can use the following template to add arguments when running a script:

Option 1
- ```
./example_script.sh arg1 arg2
```
Option 2
- ```
sh example_script.sh "arg1" "arg2"
```

Print all arguments

Write a script
- File: example_script.sh
- ```
#!/bin/sh
echo $@
```
Give permissions
- ```
chmod +x ./example_script.sh
```
Run the script with arguments
- ```
sh example_script.sh arg1 arg2
```

Iterate over arguments

Write the script

  # iterate_script.sh
  #!/bin/bash 
  for i; do 
    echo $i 
  done

Give permissions
- ```
chmod +x ./iterate_script.sh
```
Run the script with arguments
- ```
sh iterate_script.sh arg1 arg2
```

The following script is equivalent

  # iterate_script.sh
  #/bin/bash 
  for i in $*; do 
    echo $i 
  done

Count lines in files given as arguments

Write the script

# count_lines.sh
#!/bin/bash
 for i in $*; do
        nlines=$(wc -l < $i)    
        echo "There are $nlines lines in $i"
done

Give permissions
- ```
chmod +x ./count_lines.sh
```
Run the script with arguments (files). Here we use the script itself as an example.
- ```
sh count_lines.sh count_lines.sh
```

Find path of a file

Write the script

# find.sh
#!/bin/bash

find / -iname $1 2> /dev/null

Run the script
- ```
sh find.sh <filename>
```

Print how many arguments are passed in a script

Write the script

# print_qty_args.sh
#!/bin/bash
echo This script was passed $# arguments

Run the script
- ```
sh print_qty_args.sh <filename>
```

Linux

Install Terraform

Here are the steps to install Terraform on Linux based on the Terraform documentation.

wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg

echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list

sudo apt update && sudo apt install terraform

Note that the Terraform documentation also covers other methods to install Terraform on Linux.

MAC

Option 1:
- Use the following command line:
  - ```
  systemsetup -setremotelogin on
```
Option 2
- Use System Preferences
- Go to System Preferences -> Sharing -> Enable Remote Login.

Find Other storage on MAC

Open Finder > Go > Go to Folder
Paste this path
- ```
~/Library/Caches
```

Sort files by size and extension on MAC

From your desktop, press Command-F.
Click This Mac.
Click the first dropdown menu field and select Other.
From the Search Attributes window
- tick File Size and File Extension.

Windows

Install Chocolatey

To install Chocolatey on Windows, we follow the official Chocolatey website instructions.

Run PowerShell as Administrator
Check if Get-ExecutionPolicy is restricted
- ```
Get-ExecutionPolicy
```
- If it is restricted, run the following command:
  - ```
  Set-ExecutionPolicy AllSigned
```

Install Chocolatey

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Note: You might need to restart PowerShell to use Chocolatey

Install Terraform with Chocolatey

Once you've installed Chocolatey on Windows, installing Terraform is as simple as can be:

Install Terraform with Chocolatey
- ```
choco install terraform
```

Find the product key

Write the following in Command Prompt (run as administrator):

wmic path SoftwareLicensingService get OA3xOriginalProductKey

Find Windows license type

Write the following in Command Prompt:

slmgr /dli

References

GNU Bash Manual - https://www.gnu.org/software/bash/manual/bash.html

ThreeFold Manual