#ip_forward

A bunch of links encountered when trying to understand the many and various places and mechanisms by which kernel parameter values in the procfs system can be changed on a CentOS/RH 7.2 system.

2009. IPv6. Points to the RFCs to see that some values in /proc/sys are ANDed and others are ORed. http://marc.info/?l=linux-kernel&m=123606366021995&w=2

2013. StackExchange. The difference between all, eth* and default. Points to previous link. http://unix.stackexchange.com/questions/90443/sysctl-proc-sys-net-ipv46-conf-whats-the-difference-between-all-defau

In ansible it may be necessary to set ignoreerrors so that the module systctl, while calling sysctl -p will procede even in the case of unknown keys https://github.com/ansible/ansible/issues/6272

This is specifically a systemd link which dives into how to created drop-ins to override other values. See also the 2015 RH Summit at which Poettering and Breard spoke for similar content.http://events.linuxfoundation.org/sites/events/files/slides/ELC_systemd_0.pdf

Bridged networking with libvirt for RH https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Network_Configuration-Bridged_networking_with_libvirt.html

2010-2015 An ordering problem arising from sysctl attempting to set values before a kernel module is loaded. https://bugzilla.redhat.com/show_bug.cgi?id=634736

2014 Same problem to do with modules. Specifically the bridge module. AdamWilliamson points out that all the other modules except the bridge module are built into the kernel, so this problem should not arise for them https://bugzilla.redhat.com/show_bug.cgi?id=1054178

2010 - 2013 This fix is now incorporated in the iptables scripts from what I can see and works well https://bugzilla.redhat.com/show_bug.cgi?id=552522

Freedesktop how to work with systemd https://www.freedesktop.org/wiki/Software/systemd/FrequentlyAskedQuestions/ especially alias psc='ps xawf -eo pid,user,cgroup,args' show a process's cgroup and hence which service the process belongs to. Lots more good stuff there for systemd.

RH Summit Poettering Breard good systemd overview video and slides http://videos.cdn.redhat.com/summit2015/presentations/12720_demystifying-systemd.pdf

Fedora project parallels between sysv and systemd commands https://fedoraproject.org/wiki/SysVinit_to_Systemd_Cheatsheet

Tracing a systemd service http://k.japko.eu/systemd-nspawn-ping-debug.html

CoreOS rightly note the "plethora" of sysctl tuning available and show how to debug and mask systemd and use modules-load.d and cloud-config to set them https://coreos.com/os/docs/latest/other-settings.html

Systemd manpage online https://www.freedesktop.org/software/systemd/man/systemd.html

Looking at systemd services started at boot in Arch https://bbs.archlinux.org/viewtopic.php?id=186244 Mostly uses listing of Wants directories.

Official freedesktop suggestions for systemd including enabling the debug-shell https://freedesktop.org/wiki/Software/systemd/Debugging/

Arch linux troubleshooting systemd. This is worth going through https://wiki.archlinux.org/index.php/Systemd#Troubleshooting

Fedora debug systemd suggests using Wants variables specifically https://fedoraproject.org/wiki/How_to_debug_Systemd_problems systemctl show -p "Wants" multi-user.target and the test functionality of /usr/lib/systemd/systemd --test --system --unit=multi-user.target

I like this approach to replacing the unit http://centosfaq.org/centos/systemd-sysctl-not-running-on-boot/

A good overview of procsys and kernel runtime parameters on centos7.2 https://www.rootusers.com/use-procsys-and-sysctl-to-modify-and-set-kernel-runtime-parameters/

SuSE users running into systemd-sysctl ordering problems https://forums.opensuse.org/showthread.php/487543-etc-sysctl-conf-not-read-on-boot-no-longer-usable

Poettering doc on order of config files for systemd-sysctl http://0pointer.de/public/systemd-man/sysctl.d.html

Fedora Magazine 5 parter on systemd is one of the best references out there https://fedoramagazine.org/systemd-unit-dependencies-and-order/

Ubuntu and Debian use an sysv init script in 2014 which does not run procps early enough leading to some problems http://serverfault.com/questions/270201/sysctl-conf-not-running-on-boot

Debian systemd debugging suggests bumping the verbosity either via kernel commandline in grub or /etc/systemd/systemd.conf. Latter is preferrable to me. https://wiki.debian.org/systemd