Developus Apparatus

An assortment of notes and utilities for software development and systems administration.

_images/wtfpl-badge-2.png

Continuous Delivery

About

Tools

Code review

Documentation generation

Job schedulers

SCM

Distributed Source Control Management systems:

History, diff & visualisation:

Web services

Git

Git is a distributed (decentralized) Source Control Management (SCM) system.

SCM is also known as:

  • Revision Control
  • Version Control System (VCS)

Features

Social platforms

Hosting

Built-in

Git - Split and shrink a repository

Ever wanted to split a repository into several parts, yet keep the corresponding commit histories?

Procedure

Note: this is a copy of a memo, it needs to be rewritten for clarity

# first, clone the original repository
git clone REPO REPO2
cd REPO2

# remove all unneeded files from this version
git filter-branch -f --prune-empty --index-filter "git rm --cached --ignore-unmatch FILES_AND_DIRS_TO_DELETE"
git gc --aggressive --prune=1day
git fsck --unreachable

# refresh the remote
git remote rm origin
git remote add origin ssh://HOST/REPO

# broforce push!
git push -f origin master

# cleanup our original repository
cd REPO
git filter-branch -f --prune-empty --index-filter "git rm --cached --ignore-unmatch OTHER_FILES_AND_DIRS_TO_DELETE"
git gc --aggressive --prune=1day
git fsck --unreachable

# broforce push!
git push -f origin master

APT - Debian & Ubuntu packages

Resources

  • Pacman Rosetta - Compendium of usual commands for the main Linux package managers (apt, dnf, pacman, yum)

Upgrading packages

aptitude - update package metadata

# update package metadata using:
# - /etc/apt/sources.list
# - /etc/apt/sources.list.d/*.list
$ aptitude update

[...]
Get: 1 http://security.debian.org jessie/updates InRelease [63.1 kB]
Ign http://ftp.debian.org jessie InRelease
Hit http://repo.saltstack.com jessie InRelease
Get: 2 http://ftp.debian.org jessie-updates InRelease [142 kB]
Get: 3 http://security.debian.org jessie/updates/main amd64 Packages [231 kB]
Hit http://repo.saltstack.com jessie/main amd64 Packages
[...]
Fetched 12.3 MB in 9s (1,334 kB/s)

Current status: 34 updates [+28].

aptitude - upgrade all packages

$ aptitude upgrade

The following packages will be upgraded:
  apt apt-utils base-files git git-man gnupg gpgv initramfs-tools libapt-inst1.5 libapt-pkg4.12 libc-bin libc-dev-bin libc6 libc6-dev libglib2.0-0 libhogweed2 libnettle4 libpam-modules libpam-modules-bin
  libpam0g libpcre3 libsndfile1 libsystemd0 libudev1 linux-libc-dev locales multiarch-support salt-common salt-minion systemd systemd-sysv tzdata tzdata-java udev
The following packages are RECOMMENDED but will NOT be installed:
  busybox busybox-static dbus gnupg-curl libpam-systemd
34 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 32.6 MB of archives. After unpacking 71.7 kB will be freed.
Do you want to continue? [Y/n/?] y
Get: 1 http://ftp.debian.org/debian/ jessie/main base-files amd64 8+deb8u4 [78.0 kB]
Get: 2 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main salt-minion all 2015.8.8+ds-2 [24.5 kB]
Get: 3 http://ftp.debian.org/debian/ jessie/main libc6-dev amd64 2.19-18+deb8u4 [2,002 kB]
Get: 4 http://repo.saltstack.com/apt/debian/8/amd64/latest/ jessie/main salt-common all 2015.8.8+ds-2 [3,139 kB]
Get: 5 http://ftp.debian.org/debian/ jessie/main libc-dev-bin amd64 2.19-18+deb8u4 [237 kB]
[...]
Fetched 32.6 MB in 16s (1,939 kB/s)
Extracting templates from packages: 100%
Preconfiguring packages ...
(Reading database ... 37422 files and directories currently installed.)
Preparing to unpack .../base-files_8+deb8u4_amd64.deb ...
Unpacking base-files (8+deb8u4) over (8+deb8u3) ...
Processing triggers for install-info (5.2.0.dfsg.1-6) ...
Processing triggers for man-db (2.7.0.2-5) ...
Setting up base-files (8+deb8u4) ...
Installing new version of config file /etc/debian_version ...
(Reading database ... 37422 files and directories currently installed.)
[...]
Processing triggers for libc-bin (2.19-18+deb8u4) ...
Processing triggers for initramfs-tools (0.120+deb8u1) ...

Current status: 0 updates [-34].

aptitude - upgrade some packages

$ aptitude upgrade git git-email git-man

The following packages will be upgraded:
  git git-email git-man
3 packages upgraded, 0 newly installed, 0 to remove and 231 not upgraded.
Need to get 3,310 kB of archives. After unpacking 311 kB will be used.
Do you want to continue? [Y/n/?] y
Get: 1 http://archive.ubuntu.com/ubuntu/ trusty-updates/main git-man all 1:1.9.1-1ubuntu0.3 [699 kB]
Get: 2 http://archive.ubuntu.com/ubuntu/ trusty-updates/main git amd64 1:1.9.1-1ubuntu0.3 [2,586 kB]
Get: 3 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe git-email all 1:1.9.1-1ubuntu0.3 [25.5 kB]
Fetched 3,310 kB in 5s (601 kB/s)
(Reading database ... 159204 files and directories currently installed.)
Preparing to unpack .../git-man_1%3a1.9.1-1ubuntu0.3_all.deb ...
Unpacking git-man (1:1.9.1-1ubuntu0.3) over (1:1.9.1-1ubuntu0.1) ...
Preparing to unpack .../git_1%3a1.9.1-1ubuntu0.3_amd64.deb ...
Unpacking git (1:1.9.1-1ubuntu0.3) over (1:1.9.1-1ubuntu0.1) ...
Preparing to unpack .../git-email_1%3a1.9.1-1ubuntu0.3_all.deb ...
Unpacking git-email (1:1.9.1-1ubuntu0.3) over (1:1.9.1-1ubuntu0.1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up git-man (1:1.9.1-1ubuntu0.3) ...
Setting up git (1:1.9.1-1ubuntu0.3) ...
Setting up git-email (1:1.9.1-1ubuntu0.3) ...

Current status: 231 updates [-3].

Querying packages

aptitude - search a package

$ aptitude search rxvt-unicode

p   rxvt-unicode                                    - RXVT-like terminal emulator with Unicode support
p   rxvt-unicode:i386                               - RXVT-like terminal emulator with Unicode support
p   rxvt-unicode-256color                           - multi-lingual terminal emulator with Unicode support for X
p   rxvt-unicode-256color:i386                      - multi-lingual terminal emulator with Unicode support for X
p   rxvt-unicode-lite                               - RXVT-like terminal emulator with basic Unicode support
p   rxvt-unicode-lite:i386                          - RXVT-like terminal emulator with basic Unicode support
p   rxvt-unicode-ml                                 - multi-lingual terminal emulator -- transitional package
p   rxvt-unicode-ml:i386                            - multi-lingual terminal emulator -- transitional package

aptitude - show package metadata

$ aptitude show rxvt-unicode-256color

Package: rxvt-unicode-256color
State: not installed
Version: 9.19-1
Priority: optional
Section: universe/x11
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Architecture: amd64
Uncompressed Size: 3,174 k
Depends: libc6 (>= 2.17), libfontconfig1 (>= 2.9.0), libgcc1 (>= 1:4.1.1), libgdk-pixbuf2.0-0 (>=
         2.22.0), libglib2.0-0 (>= 2.12.0), libperl5.18 (>= 5.18.1), libstartup-notification0 (>= 0.2),
         libx11-6, libxft2 (> 2.1.1), libxrender1, base-passwd (>= 2.0.3.4), ncurses-term (>= 5.8-1)
Recommends: ttf-dejavu, fonts-vlgothic | fonts-japanese-gothic
Conflicts: rxvt-unicode, rxvt-unicode, rxvt-unicode-256color
Provides: rxvt-unicode, x-terminal-emulator
Description: multi-lingual terminal emulator with Unicode support for X11

Homepage: http://software.schmorp.de/pkg/rxvt-unicode.html

dpkg - search installed packages

$ dpkg --list *udev*

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                 Version         Architecture    Description
+++-====================-===============-===============-=============================================
ii  libgudev-1.0-0:amd64 1:204-5ubuntu20 amd64           GObject-based wrapper library for libudev
un  libudev0             <none>          <none>          (no description available)
ii  libudev1:amd64       204-5ubuntu20.1 amd64           libudev shared library
ii  libudev1:i386        204-5ubuntu20.1 i386            libudev shared library
ii  udev                 204-5ubuntu20.1 amd64           /dev/ and hotplug management daemon

dpkg - show local package metadata

$ dpkg --info /var/cache/apt/archives/libudev1_204-5ubuntu20.18_amd64.deb

new debian package, version 2.0.
size 33514 bytes: control archive=1549 bytes.
    612 bytes,    15 lines      control
    216 bytes,     3 lines      md5sums
    135 bytes,     7 lines   *  postinst             #!/bin/sh
    132 bytes,     7 lines   *  postrm               #!/bin/sh
     49 bytes,     2 lines      shlibs
   3982 bytes,    93 lines      symbols
Package: libudev1
Source: systemd
Version: 204-5ubuntu20.18
Architecture: amd64
Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
Installed-Size: 133
Pre-Depends: multiarch-support
Depends: libc6 (>= 2.17), libcgmanager0, libdbus-1-3 (>= 1.0.2), libnih-dbus1 (>= 1.0.0), libnih1 (>= 1.0.0)
Section: libs
Priority: important
Multi-Arch: same
Homepage: http://www.freedesktop.org/wiki/Software/systemd
Description: libudev shared library
 This library provides access to udev device information.
Original-Maintainer: Debian systemd Maintainers <pkg-systemd-maintainers@lists.alioth.debian.org>

Environment variables

Some places to look at to define/change variables, and which contents to expect.

Global

/etc/environment
global variables: HTTP(S) & SOCKS proxies, custom PATH

Session

/etc/profile
default session variables
/etc/profile.d/*.sh
extra session variables
~/.xprofile
user-defined X11 settings -mostly for lightweight session managers (i3, awesome-wm, etc.)
~/.profile
user-defined variables -superseded by ~/.bash_profile

Bash

/etc/bash.bashrc
default Bash settings
~/.bash_profile
user-defined Bash settings -mostly for X11/startup stuff
~/.bashrc
user-defined Bash configuration (main file)
~/.bash_aliases
user-defined Bash command aliases and functions

Superusers

/etc/login.defs
console & login configuration
/etc/sudoers
sudoers configuration -always use visudo to edit!

SSH

/etc/ssh/sshd_config
server-side, can allow the client to pass variables
/etc/ssh/ssh_config
client-side, can send variables to servers

SaltStack

/etc/default/salt-minion (deprecated in recent versions)
can be used to source /etc/environment -not always included in the distro’s packages
/etc/salt/minion
HTTP(S) proxy configuration

Filesystem

Disk usage

df - filesystem usage

$ df -h

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       231G   73G  147G  34% /
udev            1.9G  4.0K  1.9G   1% /dev
tmpfs           389M  972K  388M   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            1.9G   30M  1.9G   2% /run/shm

df - inode usage

$ df -hi

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda1         15M  2.6M   13M   18% /
none             486K     2  486K    1% /sys/fs/cgroup
udev             483K   564  483K    1% /dev
tmpfs            486K   594  485K    1% /run
none             486K     3  486K    1% /run/lock
none             486K     4  486K    1% /run/shm
none             486K    22  486K    1% /run/user

Partitions

fdisk - list mounted partitions

$ fdisk -l

Disk /dev/sda: 256.1 GB, 256060514304 bytes
255 heads, 63 sectors/track, 31130 cylinders, total 500118192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000681e1
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048   491870207   245934080   83  Linux
/dev/sda2       491872254   500117503     4122625    5  Extended
/dev/sda5       491872256   500117503     4122624   82  Linux swap / Solaris

lsblk - list mounted partitions

$ lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 238.5G  0 disk
├─sda1   8:1    0 234.6G  0 part /
├─sda2   8:2    0     1K  0 part
└─sda5   8:5    0     4G  0 part [SWAP]
sr0     11:0    1  1024M  0 rom

parted - resize a partition

See Using Parted.

$ parted [VOLUME]

(parted) select /dev/sda
(parted) unit GB
(parted) print
Model: ATA MTFDDAK256MAM-1K (scsi)
Disk /dev/sda: 256GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number  Start   End    Size    Type      File system     Flags
 1      0.00GB  252GB  252GB   primary   ext4            boot
 2      252GB   256GB  4.22GB  extended
 5      252GB   256GB  4.22GB  logical   linux-swap(v1)
(parted) resize 5
Start?  [252GB]? 252
End?  [256GB]? 256

Disk maintenance

See:

e2fsck error checking - manual approval

$ e2fsck /dev/sda2

e2fsck 1.42.9 (4-Feb-2014)
/dev/sda2 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 7484927 has imagic flag set.  Clear<y>? yes
Inode 7484927 has compression flag set on filesystem without compression support.  Clear<y>? yes
Inode 7484927 has INDEX_FL flag set but is not a directory.
Clear HTree index<y>? yes
[...]
/dev/sda2: ***** FILE SYSTEM WAS MODIFIED *****
/dev/sda2: 3782336/54214656 files (0.2% non-contiguous), 86369865/216833920 blocks
root@aaron:~# e2fsck -p /dev/sda2
/dev/sda2: clean, 3782336/54214656 files, 86369865/216833920 blocks

e2fsck error checking - auto approval

$ e2fsck -y /dev/sda2
/dev/sda2: clean, 3782336/54214656 files, 86369865/216833920 blocks

forced inode auto check & optimization

$ e2fsck -p -f -D /dev/sda2
/dev/sda2: 3782336/54214656 files (0.2% non-contiguous), 86367938/216833920 blocks

Networking

DNS records

Common record types

See the list of DNS record types for more information.

Type Description Function
A Address Maps a hostname to an IP
CNAME Canonical Name Alias to another hostname
MX Mail eXchange Maps a domain to a mail transfer agent
SRV SeRVice locator Generalized record type

dig

$ dig freebsd.org

; <<>> DiG 9.11.0-P1 <<>> freebsd.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1134
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;freebsd.org.                        IN      A

;; ANSWER SECTION:
freebsd.org.         3600    IN      A       8.8.178.110

;; Query time: 58 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: lun. déc. 05 19:36:05 CET 2016
;; MSG SIZE  rcvd: 45

$ dig +nocmd +noquestion +nostats archlinux.org

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25601
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; ANSWER SECTION:
archlinux.org.       1365    IN      A       138.201.81.199

$ dig +short debian.org

140.211.15.34
130.89.148.14
128.31.0.62

nslookup

$ nslookup freebsd.org

Server:      192.168.1.1
Address:     192.168.1.1#53

Non-authoritative answer:
Name:        freebsd.org
Address:     8.8.178.110
Name:        freebsd.org
Address:     2001:1900:2254:206a::50:0

Local lookup, e.g. when using dnsmasq to cache DNS results:

$ nslookup linux.org localhost

Server:      localhost
Address:     ::1#53

Non-authoritative answer:
Name:        linux.org
Address:     192.243.104.10

Port scanning - Nmap / Zenmap

State Meaning
closed the port is open, but no service is running on the remote server
filtered the port is blocked by a firewall
open the port is open, and there is a service running

nmap - open

$ nmap gerrit.example.com -p 29418 -Pn

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-07 15:46 CEST
Nmap scan report for gerrit.example.com (163.33.26.149)
Host is up (0.072s latency).
rDNS record for 163.33.26.149: irsgerrit001.ir.example.com
PORT      STATE SERVICE
29418/tcp open  unknown

Nmap done: 1 IP address (1 host up) scanned in 0.17 seconds

nmap - filtered

$ nmap gerrit.dev.example.com -p 29418 -Pn

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-07 15:50 CEST
Nmap scan report for gerrit.dev.example.com (10.96.8.73)
Host is up.
rDNS record for 10.96.8.73: fmygit6003.fm.example.com
PORT      STATE    SERVICE
29418/tcp filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.11 seconds

nmap - scan several ports, and get information on the remote services

$ nmap server.domain.example.com -p 22,80,5432,8080 -Pn -sV

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-07 16:01 CEST
Nmap scan report for server.domain.example.com (10.237.188.47)
Host is up (0.062s latency).
rDNS record for 10.225.127.36: server.domain.example.com
PORT     STATE  SERVICE    VERSION
22/tcp   open   ssh        OpenSSH 5.9p1 Debian 5ubuntu1.7 (Ubuntu Linux; protocol 2.0)
80/tcp   open   http       nginx 1.1.19
5432/tcp closed postgresql
8080/tcp open   tcpwrapped
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.52 seconds

Routing

traceroute - show routes between two hosts

$ traceroute example.com

traceroute to example.com (166.70.10.23), 30 hops max, 60 byte packets
 1  176.221.87.1 (176.221.87.1)  1.474 ms  1.444 ms  1.390 ms
 2  f126.broadband2.quicknet.se (92.43.37.126)  10.047 ms  19.868 ms  23.156 ms
 3  10.5.12.1 (10.5.12.1)  24.098 ms  24.340 ms  25.311 ms
 4  212.247.178.9 (212.247.178.9)  25.777 ms  27.184 ms  27.625 ms
 5  vst-ncore-1.bundle-ether1.tele2.net (130.244.39.46)  30.632 ms  31.610 ms  32.194 ms
 6  kst5-core-1.bundle-ether6.tele2.net (130.244.71.178)  33.608 ms  15.274 ms  16.449 ms
 7  kst5-peer-1.ae0-unit0.tele2.net (130.244.205.125) 252.53 ms 11.169 ms 12.158 ms
 8  avk6-peer-1.ae0-unit0.tele2.net (130.244.64.71)  19.661 ms  25.765 ms  26.730 ms
 9  peer-as3257.avk6.tele2.net (130.244.200.106)  25.390 ms  24.863 ms xe-5-0-0.nyc30.ip4.tinet.net (89.149.181.109)  23.626 ms
10  fortress-gw.ip4.tinet.net (216.221.158.90)  29.943 ms  31.112 ms  29.002 ms
11  208.116.63.254 (208.116.63.254)  32.102 ms  29.862 ms  29.337 ms

iptables - show local routes

$ sudo iptables -L

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-ISOLATION  all  --  anywhere             anywhere
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination

Chain DOCKER-ISOLATION (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Sockets

lsof - list active sockets

$ lsof -Pnl +M -i

rpcbind    1192        0    8u  IPv4    28745      0t0  TCP *:111[portmapper] (LISTEN)
rpcbind    1192        0   11u  IPv6    28748      0t0  TCP *:111[portmapper] (LISTEN)
cupsd      1221        0    8u  IPv6 16414694      0t0  TCP [::1]:631 (LISTEN)
rpc.statd  1238      116   11u  IPv6    11496      0t0  TCP *:55536 (LISTEN)
sshd       1295        0    3r  IPv4     1511      0t0  TCP *:22 (LISTEN)
ypbind     1395        0    5u  IPv4    28818      0t0  TCP *:724[ypbind] (LISTEN)
nrpe       1687      119    4u  IPv4    28924      0t0  TCP *:5666 (LISTEN)
nginx      1715        0   10u  IPv4     1720      0t0  TCP *:80 (LISTEN)

Processes

See http://www.binarytides.com/linux-ps-command/ for more examples!

Running

top - show resource usage & most consuming processes

$ top

htop - show resource usage & most consuming processes

$ htop

ps - list all running processes

$ ps -ef    # *nix style

$ ps aux    # BSD style

ps - process tree

$ ps faux

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    09:02   0:00 [kthreadd]
root         3  0.0  0.0      0     0 ?        S    09:02   0:11  \_ [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S    09:02   0:41  \_ [ktimersoftd/0]
root         6  0.0  0.0      0     0 ?        S<   09:02   0:00  \_ [kworker/0:0H]
root         8  0.0  0.0      0     0 ?        S    09:02   0:16  \_ [rcu_preempt]
root         9  0.0  0.0      0     0 ?        S    09:02   0:00  \_ [rcu_sched]

ps - list by user

$ ps -f -u http

UID        PID  PPID  C STIME TTY          TIME CMD
http      2546  2451  0 09:03 ?        00:00:00 /usr/bin/httpd -k start -DFOREGROUND
http      2550  2451  0 09:03 ?        00:00:00 /usr/bin/httpd -k start -DFOREGROUND
http      2552  2451  0 09:03 ?        00:00:00 /usr/bin/httpd -k start -DFOREGROUND

ps - list by process name

$ ps -C python3

  PID TTY          TIME CMD
18667 pts/3    00:00:01 python3

pgrep - find processes by name

$ pgrep -a watchdog

10 watchdog/0
11 watchdog/1
16 watchdog/2
21 watchdog/3

Zombies and defuncts

ps - display zombie processes’ PIDs

$ ps aux | awk '{ print $8 " " $2 }' | grep -w Z

ps - display a family tree (in case we’ve a defunct parent/child job)

$ ps -aef

Hardware - Serial port (RS-232)

Device baud rate

stty - read a value

$ stty -F /dev/ttyUSB0

speed 9600 baud; line = 0;
min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost -onlcr
-isig -icanon -echo

stty - set a value

$ stty -F /dev/ttyUSB0 115200

speed 115200 baud; line = 0;
min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost -onlcr
-isig -icanon -echo

Hardware - udev rules

Udev rules are located under /etc/udev/rules.d. A rule matches a set of devices, according to their vendor and product IDs, and allows to set attributes that will be applied when the device is plugged:

  • device ownership (group/user)
  • read/write permissions
  • additional mount points
  • …

Usage

Reload rules

$ udevadm control --reload-rules

Trigger device detection

$ udevadm trigger

PostgreSQL

Basics

psql - Command-Line Interface (CLI)

psql - start
root@ic-tpl:~$ su - postgres
postgres@ic-tpl:~$ psql
psql (9.4.3)
Type "help" for help.

postgres=#
psql - SQL help
# get help on SQL instructions
postgres=# \h
Available help:
  ABORT                            ALTER TYPE                       CREATE SCHEMA
  ALTER AGGREGATE                  ALTER USER                       CREATE SEQUENCE
  [...]

postgres=# \h DROP TABLE
Command:     DROP TABLE
Description: remove a table
Syntax:
DROP TABLE [ IF EXISTS ] name [, ...] [ CASCADE | RESTRICT ]
psql - Get help on the CLI (basckslash commands)
postgres=# \?

General
  \copyright             show PostgreSQL usage and distribution terms
  \g [FILE] or ;         execute query (and send results to file or |pipe)
  \gset [PREFIX]         execute query and store results in psql variables
  \h [NAME]              help on syntax of SQL commands, * for all commands
  \q                     quit psql
  \watch [SEC]           execute query every SEC seconds
Query Buffer
  \e [FILE] [LINE]       edit the query buffer (or file) with external editor
  \ef [FUNCNAME [LINE]]  edit function definition with external editor
  \p                     show the contents of the query buffer
  \r                     reset (clear) the query buffer
  \s [FILE]              display history or save it to file
  \w FILE                write query buffer to file
  [...]

Users and roles

Databases

List databases
postgres=# \l
                                     List of databases
      Name       |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------------+----------+----------+-------------+-------------+-----------------------
 icinga2_ido     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 postgres        | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
 template0       | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                 |          |          |             |             | postgres=CTc/postgres
 template1       | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
                 |          |          |             |             | postgres=CTc/postgres
(4 rows)

# the information is held by global PG tables, and can be obtained through SQL queries
postgres=# SELECT datname, datcollate FROM pg_database ORDER BY datname;
     datname     | datcollate
-----------------+-------------
 icinga2_ido     | en_US.UTF-8
 postgres        | en_US.UTF-8
 template0       | en_US.UTF-8
 template1       | en_US.UTF-8
(4 rows)
Connect to a database
postgres=# \c icinga2_ido
You are now connected to database "icinga2_ido" as user "postgres".
List tables
postgres=# \dt
                           List of relations
 Schema |                  Name                  | Type  |    Owner
--------+----------------------------------------+-------+-------------
 public | icinga_acknowledgements                | table | icinga2_ido
 public | icinga_commands                        | table | icinga2_ido
 public | icinga_commenthistory                  | table | icinga2_ido
 public | icinga_comments                        | table | icinga2_ido
Describe table
icinga2_ido=# \d icinga_commands
                                Table "public.icinga_commands"
    Column    |  Type   |                              Modifiers
--------------+---------+----------------------------------------------------------------------
 command_id   | bigint  | not null default nextval('icinga_commands_command_id_seq'::regclass)
 instance_id  | bigint  | default 0
 config_type  | integer | default 0
 object_id    | bigint  | default 0
 command_line | text    | default ''::text
Indexes:
    "pk_command_id" PRIMARY KEY, btree (command_id)
    "uq_commands" UNIQUE CONSTRAINT, btree (instance_id, object_id, config_type)
    "command_object_idx" btree (object_id)
    "commands_i_id_idx" btree (instance_id)

systemd

systemctl - daemon management

Note

Some distributions (Debian, Ubuntu & derivatives) automatically enable and start daemons after they have been installed, whereas other ones let the user manage which units are enabled and started (Archlinux, CentOS).

enable

Creates a symlink from /etc/systemd/system/<my_service> to /lib/systemd/system/<my_service> so my_service will be launched after booting.

$ root@icinga:~# systemctl enable ssh.service

Synchronizing state for ssh.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d ssh defaults
Executing /usr/sbin/update-rc.d ssh enable
Created symlink from /etc/systemd/system/sshd.service to /lib/systemd/system/ssh.service.

disable

$ root@icinga:~# systemctl disable ssh.service

Synchronizing state for ssh.service with sysvinit using update-rc.d...
Executing /usr/sbin/update-rc.d ssh defaults
Executing /usr/sbin/update-rc.d ssh disable
Removed symlink /etc/systemd/system/sshd.service.

start / stop / restart

$ root@icinga:~# systemctl start php5-fpm.service

$ root@icinga:~# systemctl stop nginx.service

$ root@icinga:~# systemctl restart nginx.service

list-units - active services

$ root@icinga:~# systemctl list-units -t service

 UNIT                             LOAD   ACTIVE SUB     DESCRIPTION
 console-getty.service            loaded active running Console Getty
 cron.service                     loaded active running Regular background program processing daemon
 exim4.service                    loaded active running LSB: exim Mail Transport Agent
 getty-static.service             loaded active exited  getty on tty2-tty6 if dbus and logind are not available
 getty@tty1.service               loaded active running Getty on tty1
 getty@tty2.service               loaded active running Getty on tty2
 getty@tty3.service               loaded active running Getty on tty3
 getty@tty4.service               loaded active running Getty on tty4
 getty@tty5.service               loaded active running Getty on tty5
 getty@tty6.service               loaded active running Getty on tty6
 icinga2.service                  loaded active running Icinga host/service/network monitoring system
 modules_dep.service              loaded active exited  LSB: modules.dep creation.
 nagios-nrpe-server.service       loaded active running LSB: Start/Stop the Nagios remote plugin execution daemon
 networking.service               loaded active running LSB: Raise network interfaces.
 nginx.service                    loaded active running A high performance web server and a reverse proxy server
 ntp.service                      loaded active exited  LSB: Start NTP daemon
 php5-fpm.service                 loaded active running The PHP FastCGI Process Manager
 postgresql.service               loaded active exited  PostgreSQL RDBMS
 postgresql@9.4-main.service      loaded active running PostgreSQL Cluster 9.4-main
 quota.service                    loaded active exited  Check And Enable File System Quotas
●rc-local.service                 loaded failed failed  /etc/rc.local Compatibility
 rsyslog.service                  loaded active running System Logging Service
 ssh.service                      loaded active running OpenBSD Secure Shell server
 systemd-journald.service         loaded active running Journal Service
 systemd-random-seed.service      loaded active exited  Load/Save Random Seed
 systemd-remount-fs.service       loaded active exited  Remount Root and Kernel File Systems
 systemd-resolved.service         loaded active running Network Name Resolution
 systemd-setup-dgram-qlen.service loaded active exited  Increase datagram queue length
 systemd-sysctl.service           loaded active exited  Apply Kernel Variables
 systemd-tmpfiles-setup.service   loaded active exited  Create Volatile Files and Directories
 systemd-udev-trigger.service     loaded active exited  udev Coldplug all Devices
 systemd-udevd.service            loaded active running udev Kernel Device Manager
 systemd-update-utmp.service      loaded active exited  Update UTMP about System Boot/Shutdown
 systemd-user-sessions.service    loaded active exited  Permit User Sessions
 udev-finish.service              loaded active exited  Copy rules generated while the root was ro

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

35 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

status

$ root@icinga:~# systemctl status php5-fpm.service

● php5-fpm.service - The PHP FastCGI Process Manager
   Loaded: loaded (/lib/systemd/system/php5-fpm.service; enabled)
   Active: active (running) since Thu 2015-09-03 11:49:59 CEST; 5h 18min ago
 Main PID: 31259 (php5-fpm)
   Status: "Processes active: 0, idle: 2, Requests: 52, slow: 0, Traffic: 0req/sec"
   CGroup: /system.slice/php5-fpm.service
           ├─31259 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
           ├─31260 php-fpm: pool www
           └─31261 php-fpm: pool www
Sep 03 11:49:59 icinga systemd[1]: Started The PHP FastCGI Process Manager.

status - units matching a pattern

$ root@icinga:~# systemctl status po*

● postgresql.service - PostgreSQL RDBMS
   Loaded: loaded (/lib/systemd/system/postgresql.service; enabled)
   Active: active (exited) since Thu 2015-09-03 11:53:06 CEST; 5h 50min ago
 Main PID: 31552 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/postgresql.service

Sep 03 11:53:06 icinga systemd[1]: Started PostgreSQL RDBMS.

● postgresql@9.4-main.service - PostgreSQL Cluster 9.4-main
   Loaded: loaded (/lib/systemd/system/postgresql@.service; disabled)
   Active: active (running) since Thu 2015-09-03 11:53:06 CEST; 5h 50min ago
 Main PID: 31520 (postgres)
   CGroup: /system.slice/system-postgresql.slice/postgresql@9.4-main.service
           ├─31520 /usr/lib/postgresql/9.4/bin/postgres -D /var/lib/postgresql/9.4/main -c config_file=/etc/postgresql/9.4/main/postgresql.conf
           ├─31522 postgres: checkpointer process
           ├─31523 postgres: writer process
           ├─31524 postgres: wal writer process
           ├─31525 postgres: autovacuum launcher process
           ├─31526 postgres: stats collector process
           └─31534 postgres: icinga2idopgsql icinga2idopgsql ::1(36874) idle in transaction

Sep 03 11:53:06 icinga systemd[1]: Started PostgreSQL Cluster 9.4-main.

journalctl

journalctl - logs for a given daemon

$ root@icinga:~# journalctl -b -u ssh.service

-- Logs begin at Tue 2015-09-01 17:27:04 CEST, end at Thu 2015-09-03 17:21:06 CEST. --
Sep 01 17:27:04 icinga systemd[1]: Starting OpenBSD Secure Shell server...
Sep 01 17:27:04 icinga systemd[1]: Started OpenBSD Secure Shell server.
Sep 01 17:27:04 icinga sshd[172]: Server listening on 0.0.0.0 port 22.
Sep 01 17:27:04 icinga sshd[172]: Server listening on :: port 22.
Sep 01 17:27:04 icinga sshd[172]: Could not load host key: /etc/ssh/ssh_host_rsa_key
Sep 01 17:27:04 icinga sshd[172]: Could not load host key: /etc/ssh/ssh_host_dsa_key
Sep 01 17:27:04 icinga sshd[172]: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Sep 01 17:27:04 icinga sshd[172]: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Sep 01 17:27:05 icinga systemd[1]: Stopping OpenBSD Secure Shell server...
Sep 01 17:27:05 icinga systemd[1]: Starting OpenBSD Secure Shell server...
Sep 01 17:27:05 icinga systemd[1]: Started OpenBSD Secure Shell server.
Sep 01 17:27:05 icinga sshd[203]: Server listening on 0.0.0.0 port 22.
Sep 01 17:27:05 icinga sshd[203]: Server listening on :: port 22.
Sep 01 17:50:24 icinga sshd[1566]: Accepted password for root from 10.102.167.30 port 39590 ssh2
Sep 01 17:50:24 icinga sshd[1566]: pam_unix(sshd:session): session opened for user root by (uid=0)

journalctl - list system boots

$ root@icinga:~# journalctl --list-boots

0 897795c2801a4197bbe425f0d6d59ce3 Tue 2015-09-01 17:27:04 CEST—Thu 2015-09-03 17:24:06 CEST

Docker - Usage

Basics

Install Docker, by following the instruction relevant to your OS / distribution, and start the service.

Search an image on DockerHub

$ docker search debian

NAME            DESCRIPTION                                     STARS   OFFICIAL   AUTOMATED
ubuntu          Ubuntu is a Debian-based Linux operating s...   2065    [OK]
debian          Debian is a Linux distribution that's comp...   603     [OK]
google/debian                                                   47                 [OK]

Show available tags for a repository

$ curl https://index.docker.io/v1/repositories/debian/tags | python -m json.tool

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  1283    0  1283    0     0    433      0 --:--:--  0:00:02 --:--:--   433

Sample output:

[
    {
        "layer": "85a02782",
        "name": "stretch"
    },
    {
        "layer": "59abecbc",
        "name": "testing"
    },
    {
        "layer": "bf0fd686",
        "name": "unstable"
    },
    {
        "layer": "60c52dbe",
        "name": "wheezy"
    },
    {
        "layer": "c5b806fe",
        "name": "wheezy-backports"
    }
]

Pull an image from DockerHub

$ docker pull repository[:tag]

$ docker pull debian:wheezy
wheezy: Pulling from debian
4c8cbfd2973e: Pull complete
60c52dbe9d91: Pull complete
Digest: sha256:c584131da2ac1948aa3e66468a4424b6aea2f33acba7cec0b631bdb56254c4fe
Status: Downloaded newer image for debian:wheezy

Run!

Get the Shaarli image

$ docker pull shaarli/shaarli
latest: Pulling from shaarli/shaarli
32716d9fcddb: Pull complete
84899d045435: Pull complete
4b6ad7444763: Pull complete
e0345ef7a3e0: Pull complete
5c1dd344094f: Pull complete
6422305a200b: Pull complete
7d63f861dbef: Pull complete
3eb97210645c: Pull complete
869319d746ff: Already exists
869319d746ff: Pulling fs layer
902b87aaaec9: Already exists
Digest: sha256:f836b4627b958b3f83f59c332f22f02fcd495ace3056f2be2c4912bd8704cc98
Status: Downloaded newer image for shaarli/shaarli:latest

Create and start a new container from the image

# map the host's :8000 port to the container's :80 port
$ docker create -p 8000:80 shaarli/shaarli
d40b7af693d678958adedfb88f87d6ea0237186c23de5c4102a55a8fcb499101

# launch the container in the background
$ docker start d40b7af693d678958adedfb88f87d6ea0237186c23de5c4102a55a8fcb499101
d40b7af693d678958adedfb88f87d6ea0237186c23de5c4102a55a8fcb499101

# list active containers
$ docker ps
CONTAINER ID  IMAGE            COMMAND               CREATED         STATUS        PORTS                 NAMES
d40b7af693d6  shaarli/shaarli  /usr/bin/supervisor  15 seconds ago  Up 4 seconds  0.0.0.0:8000->80/tcp  backstabbing_galileo

Stop and destroy a container

$ docker stop backstabbing_galileo  # those docker guys are really rude to physicists!
backstabbing_galileo

# check the container is stopped
$ docker ps
CONTAINER ID  IMAGE            COMMAND               CREATED         STATUS        PORTS                 NAMES

# list ALL containers
$ docker ps -a
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS                      PORTS               NAMES
d40b7af693d6        shaarli/shaarli     /usr/bin/supervisor   5 minutes ago       Exited (0) 48 seconds ago                       backstabbing_galileo

# destroy the container
$ docker rm backstabbing_galileo  # let's put an end to these barbarian practices
backstabbing_galileo

$ docker ps -a
CONTAINER ID  IMAGE            COMMAND               CREATED         STATUS        PORTS                 NAMES

LXC

About

Linux templates

A template is a script allowing to populate a container’s pseudo-filesystem for it to run a given Linux distribution.

To display a template’s options:

$ lxc-create -t <template> -h

For more information:

$ emacs /usr/share/lxc/templates/lxc-<template>

Useful commands

List existing containers and their network address (if running):

$ lxc-ls --fancy

NAME                STATE   AUTOSTART GROUPS IPV4       IPV6
example-centos-test RUNNING 0         -      10.0.3.176 -
example-ubuntu-test RUNNING 0         -      10.0.3.179 -

Container lifecycle

CentOS 7

Prerequisite: yum

$ lxc-create -t centos -n example-centos-test -- -R 7
$ lxc-start -n example-centos-test
$ lxc-attach -n example-centos-test
# do something with the container
$ lxc-stop -n example-centos-test
$ lxc-destroy -n example-centos-test

Debian 8

Prerequisite: debootstrap

$ lxc-create -t debian -n example-debian-test
$ lxc-start -n example-debian-test
$ lxc-attach -n example-debian-test
# do something with the container
$ lxc-stop -n example-debian-test
$ lxc-destroy -n example-debian-test

Ubuntu 16.04

Prerequisite: debootstrap (recent version)

$ lxc-create -t ubuntu -n example-ubuntu-test -- -r xenial
$ lxc-start -n example-ubuntu-test
$ lxc-attach -n example-ubuntu-test
# do something with the container
$ lxc-stop -n example-ubuntu-test
$ lxc-destroy -n example-ubuntu-test

LXC - Debian installation

Packages

Let’s follow the Debian LXC guide!

$ aptitude update

# let's get a decent LXC version ;-)
$ aptitude -t jessie-backports install lxc

# required for recent Debian/Ubuntu containers
$ aptitude -t jessie-backports install debootstrap

# required for CentOS containers
$ aptitude install yum

Network configuration

/etc/default/lxc-net
USE_LXC_BRIDGE="true"
/etc/lxc/default.conf
lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx

Applying the modifications

Restart lxc-net so the modifications are taken into account, and the lxcbr0 network bridge is created:

$ systemctl restart lxc-net

$ ip -4 addr show lxcbr0
5: lxcbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    inet 10.0.3.1/24 scope global lxcbr0
       valid_lft forever preferred_lft forever

OpenVZ

Scripts

mkvztpl.sh

Creates an OpenVZ template from an existing container instance.

mkvztpl.sh
#!/bin/bash
#
# Creates an OpenVZ template from a container instance
#
# $1 ID of the container to use for template creation
# $2 Name of the template, without extension
ID=${1}
NAME=${2}

CUR_DIR=${PWD}

VZ_DIR=/var/lib/vz
ROOT_DIR=${VZ_DIR}/root/${ID}
TPL=${VZ_DIR}/template/cache/${NAME}.tar.gz

echo "> ${ID}: Cleaning package cache"
sudo vzctl restart ${ID}
sudo vzctl exec ${ID} apt-get clean
sudo vzctl stop ${ID}

echo "> ${ID}: Mounting filesystem"
sudo vzctl mount ${ID}

echo "> ${ID}: Cleaning up configuration"
cd ${ROOT_DIR}/etc
sudo rm -f hostname resolv.conf
sudo cp rc.local.first rc.local

cd ssh
sudo rm -f ssh_host_*

echo "> ${ID}: Archiving to ${TPL}"
cd ${ROOT_DIR}
sudo tar --numeric-owner -zcf ${TPL} .

echo "> ${ID}: Unmounting filesystem"
cd ${CUR_DIR}
sudo vzctl umount ${ID}

Usage:

$ ./mkvztpl.sh <CT_ID> <TPL_NAME>

Java

Look & Feel

Add to ~/.profile or ~/.bashrc:

export _JAVA_OPTIONS='-Dswing.defaultlaf=com.sun.java.swing.plaf.gtk.GTKLookAndFeel'

Python

Basics

Packages

Virtualenv

  • virtualenv - creates virtual environments (PyPi - code - doc)
  • virtualenvwrapper - manages projects and virtual environments (PyPi - code - doc)
  • pew - Python Env Wrapper (PyPi - code)

Code quality

Static Analysis

  • isort - sorts imported packages and modules (PyPi - code)
  • pep8 - checks some of the style conventions in PEP 8 (PyPi - code - doc)
  • pylint - checks for errors, tries to enforce a coding standard and looks for bad code smells (PyPi - code - doc)

Test frameworks

Useful packages

Networking

Science

  • IPython - interactive Python shell (PyPi - code - doc)
  • Jupyter Notebook (formerly IPython Notebook) (PyPi - code - doc)
  • Numpy - N-dimensional array manipulation (PyPi - code - doc)
  • Scipy - mathematics, science, and engineering (PyPi - code - doc)

SCM

  • Dulwich - native implementation of Git in Python (PyPi - code - doc)
  • GitPython - Git wrapper (PyPi - code - doc)
    • note: the documentation is very scarce, delving into the code is required to understand object relationship

Service management

Templating

Web

WSGI

Elastic Stack

Overview

The Elastic Stack, formerly known as ELK, is a software suite composed of the following core components:

A typical installation might also feature:

  • the X-Pack security, machine learning and monitoring plugin collection;
  • several Beats lightweight data shippers.

Setup

Security

The X-Pack plugin comes with a subscription plan and a 30-day trial license:

Once the license has expired, a number of features become unavailable, among which is user management (authentication, authorization).

This limitation can be circumvented by serving the Elastic Stack services behind a reverse HTTP proxy, using Basic Authentication features to manage user authentication and provide simple authorization:

Tutorials and examples

Python bindings