Random sysop stuff

Storing multiline string into GITHUB_ENV

Trying to store multiline string into GITHUB_ENV will fail with

Error: Unable to process file command 'env' successfully.

Working workaround:

Rebasing one commit over another with git

Git rebase has a handy option for transplanting a particular commit over another. It’s called “--onto” and in contrast to rebasing alone has the capability to rebase in a more granular manner, i.e. rebase particular commit only and not the whole branch which is the case with rebase by default.

An excerpt from git manual shows the transition:

Before rebasing with “--onto”:

o---o---o---o---o  master         \           o---o---o---o---o  next                                   \                                    o---o---o  topic

After:

o---o---o---o---o  master       |                 \       |                  o'--o'--o'  topic        \          o---o---o---o---o  next

The actual command:

git rebase \-\-onto master next topic

Unstaging commited files in GIT

Unstage all files from most recent commit:

git reset --soft HEAD~1

SSH access to Azure DevOps repository

By default cloning a repo hosted at Azure DevOps will fail with:

Unable to negotiate with <Azure IP> port 22: no matching host key type found. Their offer: ssh-rsa. fatal: Could not read from remote repository.

Azure DevOps is offering to authenticate with ssh-rsa which is considered insecure and was deprecated in OpenSSH ver. 8.8.

To override this a workaround is required by adding following entries to ~/.ssh/config:

Host ssh.dev.azure.com   HostkeyAlgorithms +ssh-rsa   PubkeyAcceptedAlgorithms +ssh-rsa

Setup debconf frontend

By default debconf uses ‘dialog’ interactive frontend so in case there are options to choose from during package installation a relevant dialog with a list of available options will pop up. Even if apt is used with -y (--assume-yes) option it still may require some user input. To completely disable this behavior and make the dpkg backend always choose the default options (i.e. in non interactive scripts) set the debconf frontend to ‘noninteractive’ by exporting relevant environment variable:

export DEBIAN_FRONTEND=noninteractive

Reconfigure default editor

sudo update-alternatives --config editor

Persisting netfilter firewall rules across boots in Ubuntu Xenial

Install iptables-persistent package.

Rules (IPv4) to be applied are held in /etc/iptables/rules.v4 file. Dump current rules to it:

iptables-save > /etc/iptables/rules.v4

Rules management is done by a native systemd service named netfilter-persistent:

#Enable rules at boot: systemctl enable netfilter-persistent #Load rules manually systemctl start netfilter-persistent #Clean all rules /usr/sbin/netfilter-persistent flush

Listing all available package versions

Rhel/CentOS

“--showduplicates“ option of yum’s “list” command shows all available versions for a given package:

[root@a3205c7126f0 /]# yum list --showduplicates python3 (...) Available Packages python3.i686                                                        3.6.8-17.el7                                                       base                      python3.x86_64                                                      3.6.8-17.el7                                                       base                      python3.i686                                                        3.6.8-18.el7                                                       updates                   python3.x86_64                                                      3.6.8-18.el7                                                       updates

Debian/Ubuntu

root@18988a2aac6d:/# apt policy tar tar:  Installed: 1.29b-2ubuntu0.2  Candidate: 1.29b-2ubuntu0.2  Version table: *** 1.29b-2ubuntu0.2 500        500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages        500 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages        100 /var/lib/dpkg/status     1.29b-2 500        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages

“***” marks the currently installed version

Installing explicit version of a package

Rhel/CentOS

Follow package name by “-” plus version string:

[root@a3205c7126f0 /]# yum install python3-3.6.8-17.el7 (...)

Debian/Ubuntu

Follow package name by “=” plus version string:

Kubectl cheat sheet

Get given context details:

# kubectl config get-contexts local CURRENT   NAME    CLUSTER      AUTHINFO           NAMESPACE *                       local       local                kube-admin-local

Set current context:

# kubectl config use-context local Switched to context "local".

Set default namespace for current context:

# kubectl config set-context --current --namespace=foo Context "local" modified. # kubectl config get-contexts local CURRENT   NAME    CLUSTER   AUTHINFO           NAMESPACE *                       local        local            kube-admin-local   foo

List pods for a given deployment:

# kubectl describe deployments.apps <deployment-name> | grep ^Selector Selector:               app=<app name> # kubectl get pods --selector app=<app name> NAME                  READY   STATUS    RESTARTS   AGE <pod name>            1/1       Running    0                   XXm

Get all pods running on specific k8s node:

Create virtualenv for specific python version

Handy Openssl CLI commands

Establishing a session to a remote server and printing out session and cert info:

openssl s_client -connect <host>:443

Print out cert info in text form:

openssl x509 -text -noout < cert.pem

List available SSL cipher list with a complete description of protocol version:

openssl ciphers -v

Python Cryptography module recipes

Printing SubjectAlternativeName extension details for a given certificate

from cryptography import x509

f = open('cert-file.crt', 'rb') x509_cert = x509.load_pem_x509_certificate(f.read()) san = x509_cert.extensions.get_extension_for_class(x509.SubjectAlternativeName) print(san.value.get_values_for_type(x509.DNSName))

Listing logged in private registries

On a technical background logging in to a private docker registry isn’t setting up any session, it’s just the process of storing registry credentials for use whenever needed.

List of registries for which there has been a login operation performed is kept in ~/.docker/config.json file:

Create box with custom disk size

By default Vagrant does not provide any feature to set custom disk size for a box. To do so vagrant-disksize plugin needs to be installed:

$ vagrant  plugin install vagrant-disksize Installing the 'vagrant-disksize' plugin. This can take a few minutes... Installed the plugin 'vagrant-disksize (0.1.3)'!

Configure disk size in Vagrantfile:

Vagrant.configure("2") do |config| (...)  config.disksize.size = '50GB' (...)

Run “vagrant up” do start vm:

$ vagrant up (...) ==> default: Resized disk: old 10240 MB, req 51200 MB, new 51200 MB ==> default: You may need to resize the filesystem from within the guest. (...)

Above instructions are valid for virtualbox provider only as mentioned plugin doesn’t support other providers. VIrtualBox version used was 5.2.32.

Redirecting Unix socket via ssh tunnel

socat “UNIX-LISTEN:/path/to/socket_on_local_side.sock,reuseaddr,fork,mode=777″ EXEC:’ssh -p <ssh port on remote> <user>@<ip of remote end> -i <ssh identity file> socat STDIO UNIX-CONNECT\:/path/to/sock_on_remote.sock‘

Instance Resource Quota

IO limits

OS uses QEMU’s blkdeviotune underneath to apply i/o limits. Limits are setup at flavor level, altering them requires resizing an instance.

Setting up flavor with limits:

nova flavor-key <flavor name> set quota:disk_read_bytes_sec=<limit> nova flavor-key <flavor name> set quota:disk_write_bytes_sec=<limit>

CPU limits

Those are handled with CPU cgroup controller and applied at flavor level as well:

nova flavor-key <flavor name> set quota:cpu_quota=<limit> nova flavor-key <flavor name> set quota:cpu_period=<limit>

Bandwidth limits

Tc is used to control traffic shaping.

nova flavor-key <flavor name> set quota:vif_inbound_average=<limit> nova flavor-key <flavor name> set quota:vif_outbound_average=<limit>

Ref:. https://wiki.openstack.org/wiki/InstanceResourceQuota

Persisting DevStack’s ‘br-ex’ interface configuration across host OS reboots (on Ubuntu 18.04)

Stack.sh

Once stack.sh successfully deploys DevStack the ‘br-ex’ OVS bridge is assigned an IPv4 address of 172.24.4.1/24:

br-ex: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000 link/ether 46:98:c5:26:1d:49 brd ff:ff:ff:ff:ff:ff inet 172.24.4.1/24 brd 172.24.4.255 scope global br-ex   valid_lft forever preferred_lft forever

Also there is an iptables rule for floating IPs masking:

# /sbin/iptables -t nat -L POSTROUTING (...) MASQUERADE  all  --  172.24.4.0/24        anywhere

Once the host OS is rebooted all of the above is lost and OS instances are no longer reachable via floating IPs assigned to them.

Persisting the configuration

Create Netplan yaml file for br-ex interface:

/# cd /etc/netplan/ /etc/netplan# cat 99-br-ex.yaml network:    version: 2    renderer: networkd    ethernets:        br-ex:            addresses:                - 172.24.4.1/24

Test new settings:

/etc/netplan# netplan try (...) Configuration accepted.

Persist iptables rule:

Manually adjusting CPUs frequencies using sysfs cpufreq interface

Sysfs interface paths shown as of kernel ver. 4.15

Adjusting scaling governor

List available governors:

# cd /sys/devices/system/cpu/cpufreq /sys/devices/system/cpu/cpufreq# cat policy?/scaling_available_governors | uniq conservative ondemand userspace powersave performance schedutil

Load userspace governor to allow manual frequency setup:

/sys/devices/system/cpu/cpufreq# echo userspace > policy0/scaling_governor /sys/devices/system/cpu/cpufreq# echo userspace > policy1/scaling_governor

Adjusting CPU frequency

List available frequencies:

/sys/devices/system/cpu/cpufreq# cat policy?/scaling_available_frequencies | uniq 2401000 2400000 2000000 1600000 1200000 800000

Set up specific freq of choice:

/sys/devices/system/cpu/cpufreq# echo 2000000 > policy0/scaling_setspeed /sys/devices/system/cpu/cpufreq# echo 2000000 > policy1/scaling_setspeed

Verify current freq was actually set:

/sys/devices/system/cpu/cpufreq# cat policy?/cpuinfo_cur_freq | uniq 2000000 /sys/devices/system/cpu/cpufreq# grep -i mhz /proc/cpuinfo cpu MHz         : 1994.998 cpu MHz         : 1994.998

Enabling fstrim on various storage layers

dm-crypt

/etc/crypttab sda1_crypt         /dev/sda1          none             luks,discard

LVM

/etc/lvm/lvm.conf (...) devices {   (...)   issue_discards = 1   (...) } (...)

Filesystem

Fstrim on any filesystem can be enabled by adding discard flag to fstab. Since it will trigger defragmentation and deletion of freed blocks on every remove operation at filesystem level, it’s highly discouraged to do so. Instead one should run fstrim command periodically. Shedule should be adjusted to a particular use case. Command to check which devices support fstrim:

#lsblk -o KNAME,DISC-MAX KNAME DISC-MAX sda         2G sda1        2G sda2        2G dm-0        2G dm-1        2G

#cat /sys/block/sda/queue/discard_max_bytes 2147450880

Those with DISC-MAX of non zero value do support fstrim.

QEMU

To leverage fstrim inside domain we have to setup disk bus to utilise SCSI and SCSI controller to VirtIO SCSI:

<controller type='scsi' index='0' model='virtio-scsi'>   <alias name='scsi0'/>    <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/> </controller> <disk type='network' device='disk'> (..)   <target dev='sda' bus='scsi'/> (...)

Then we add discard='unmap' option to disk definition:

<driver name='qemu' type='raw' cache='writeback' discard='unmap'/>

Creating LVM thin pool and thin pool volumes

At first, install thin_check tool, eg. in Ubuntu run:

apt install thin-provisioning-tools

Create optimally aligned partition for physical volume using parted:

# parted -a optimal /dev/sdb mkpart pri 0% 200G

# parted /dev/sdb align-check optimal 1 1 aligned

Create LVM physical volume and volume group:

# pvcreate /dev/sdb1  Physical volume "/dev/sdb1" successfully created.

# vgcreate vg1 /dev/sdb1  Volume group "vg1" successfully created

Create thin pool volume for thin volumes allocation:

# lvcreate --type thin-pool --extents 47635 --name  vg1-thin-pool vg1  Using default stripesize 64,00 KiB.  Thin pool volume with chunk size 128,00 KiB can address at most 31,62 TiB of data.  Logical volume "vg1-thin-pool" created.

Finally create logical volume withing previously created thin pool:

Download Windows XP Mode executable from:

https://www.microsoft.com/en-us/download/details.aspx?id=8002

The main idea is to extract Windows XP virtual drive image from archive and launch it in VirtualBox.

Install p7zip-full:

apt-get install p7zip-full

Extract virtual drive using following commands:

7z x WindowsXPMode_pl-pl.exe cd sources/ 7z x xpm mv VirtualXPVHD VirtualXP.VHD

Install VirtualBox:

apt-get install virtualbox

Setup a P2P mode on Ubuntu 16.04 with PSK static key

Install

apt-get update apt-get install openvpn

Generate a shared secret key and put it into file:

openvpn --genkey --secret /etc/openvpn/secret.key

Enforce safe secret key file mode:

chmod 0600 /etc/openvpn/secret.key

Deploy daemon configuration

Note custom cipher - it defaults to BF-CBC which is deprecated due to block size of less than 128 bits. Display all available ciphers with:

openvpn --show-ciphers

/etc/default/openvpn

AUTOSTART="all"

Issue a 'systemctl daemon-reload' command after modifying above file.

/etc/openvpn/server.conf

cipher AES-256-CBC port 1194 proto udp dev tun secret secret.key ifconfig 10.8.0.1 10.8.0.2 keepalive 10 120 comp-lzo user nobody group nogroup persist-key persist-tun status openvpn-status.log verb 3

Peer's configuration

/etc/openvpn/<vpn_name_of_your_choice>.conf

remote <remote peer's IP address> 1194 udp secret secret.key dev tun ifconfig 10.8.0.2 10.8.0.1 cipher AES-256-CBC comp-lzo

Put shared key in /etc/openvpn/secret.key and:

chmod 0600 /etc/openvpn/secret.key

Start server and client

Php-fpm directives

pm.start_servers – number of PHP-FPM children spawned automatically

pm.max_children – maximum number of children allowed

pm.min_spare_servers – minimum number of spare idle PHP-FPM servers to have available

pm.max_spare_servers – maximum number of spare idle PHP-FPM servers to have available

pm.max_requests – max number of requests each child should handle before re-spawning

pm.request_terminate_timeout – Maximum amount of time to process a request

Setup custom disk layout on a cloud image with cloud-init

By default cloud-init grows pre build cloud image to span the whole underlying disk image. It uses “Growpart” module for that purpose and disabling it’s actions requires custom module setting:

growpart:   mode: off

Cloud instance’s disk image will now have unresized root partition and unallocated free space which can be used to setup additional partitions. Rootfs needs to be grown to some sane size manually.

Usage of ‘frontend/backend’ proxy directive versus ‘listen’

Frontend/Backend:

   # Simple configuration for an HTTP proxy listening on port 80 on all    # interfaces and forwarding requests to a single backend "servers" with a    # single server "server1" listening on 127.0.0.1:8000    global        daemon        maxconn 256

   defaults        mode http        timeout connect 5000ms        timeout client 50000ms        timeout server 50000ms

   frontend http-in        bind *:80        default_backend servers

   backend servers        server server1 127.0.0.1:8000 maxconn 32

‘Listen’ equivalent:

   # The same configuration defined with a single listen block. Shorter but    # less expressive, especially in HTTP mode.    global        daemon        maxconn 256

   defaults        mode http        timeout connect 5000ms        timeout client 50000ms        timeout server 50000ms

   listen http-in        bind *:80        server server1 127.0.0.1:8000 maxconn 32