Fun with sysctl, systemd, libvirt and forwarding
A bunch of links encountered when trying to understand the many and various places and mechanisms by which kernel parameter values in the procfs system can be changed on a CentOS/RH 7.2 system.
2009. IPv6. Points to the RFCs to see that some values in /proc/sys are ANDed and others are ORed. http://marc.info/?l=linux-kernel&m=123606366021995&w=2
2013. StackExchange. The difference between all, eth* and default. Points to previous link. http://unix.stackexchange.com/questions/90443/sysctl-proc-sys-net-ipv46-conf-whats-the-difference-between-all-defau
In ansible it may be necessary to set ignoreerrors so that the module systctl, while calling sysctl -p will procede even in the case of unknown keys https://github.com/ansible/ansible/issues/6272
This is specifically a systemd link which dives into how to created drop-ins to override other values. See also the 2015 RH Summit at which Poettering and Breard spoke for similar content.http://events.linuxfoundation.org/sites/events/files/slides/ELC_systemd_0.pdf
Bridged networking with libvirt for RH https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Network_Configuration-Bridged_networking_with_libvirt.html
2010-2015 An ordering problem arising from sysctl attempting to set values before a kernel module is loaded. https://bugzilla.redhat.com/show_bug.cgi?id=634736
2014 Same problem to do with modules. Specifically the bridge module. AdamWilliamson points out that all the other modules except the bridge module are built into the kernel, so this problem should not arise for them https://bugzilla.redhat.com/show_bug.cgi?id=1054178
2010 - 2013 This fix is now incorporated in the iptables scripts from what I can see and works well https://bugzilla.redhat.com/show_bug.cgi?id=552522
Freedesktop how to work with systemd https://www.freedesktop.org/wiki/Software/systemd/FrequentlyAskedQuestions/ especially alias psc='ps xawf -eo pid,user,cgroup,args' show a process's cgroup and hence which service the process belongs to. Lots more good stuff there for systemd.
RH Summit Poettering Breard good systemd overview video and slides http://videos.cdn.redhat.com/summit2015/presentations/12720_demystifying-systemd.pdf
Fedora project parallels between sysv and systemd commands https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet
Tracing a systemd service http://k.japko.eu/systemd-nspawn-ping-debug.html
CoreOS rightly note the "plethora" of sysctl tuning available and show how to debug and mask systemd and use modules-load.d and cloud-config to set them https://coreos.com/os/docs/latest/other-settings.html
Systemd manpage online https://www.freedesktop.org/software/systemd/man/systemd.html
Looking at systemd services started at boot in Arch https://bbs.archlinux.org/viewtopic.php?id=186244 Mostly uses listing of Wants directories.
Official freedesktop suggestions for systemd including enabling the debug-shell https://freedesktop.org/wiki/Software/systemd/Debugging/
Arch linux troubleshooting systemd. This is worth going through https://wiki.archlinux.org/index.php/Systemd#Troubleshooting
Fedora debug systemd suggests using Wants variables specifically https://fedoraproject.org/wiki/How_to_debug_Systemd_problems systemctl show -p "Wants" multi-user.target and the test functionality of /usr/lib/systemd/systemd --test --system --unit=multi-user.target
I like this approach to replacing the unit http://centosfaq.org/centos/systemd-sysctl-not-running-on-boot/
A good overview of procsys and kernel runtime parameters on centos7.2 https://www.rootusers.com/use-procsys-and-sysctl-to-modify-and-set-kernel-runtime-parameters/
SuSE users running into systemd-sysctl ordering problems https://forums.opensuse.org/showthread.php/487543-etc-sysctl-conf-not-read-on-boot-no-longer-usable
Poettering doc on order of config files for systemd-sysctl http://0pointer.de/public/systemd-man/sysctl.d.html
Fedora Magazine 5 parter on systemd is one of the best references out there https://fedoramagazine.org/systemd-unit-dependencies-and-order/
Ubuntu and Debian use an sysv init script in 2014 which does not run procps early enough leading to some problems http://serverfault.com/questions/270201/sysctl-conf-not-running-on-boot
Debian systemd debugging suggests bumping the verbosity either via kernel commandline in grub or /etc/systemd/systemd.conf. Latter is preferrable to me. https://wiki.debian.org/systemd
The always indispensable Dell KVM guide to libvirt on RHEL. This is one of the best resources http://linux.dell.com/files/whitepapers/KVM_Virtualization_in_RHEL_7_Made_Easy.pdf