#cfengine

I've been having trouble adding a second host to my CFEngine network. It looks like out-of-the-box network security in CentOS (which is the O/S running on my policy server) is preventing the new Ubuntu host from connecting and downloading policy.

I checked to see if iptables was running on the policy server:

# /etc/init.d/iptables status (yep, plenty of host firewall rules being enforced!) # chkconfig iptables && echo "iptables set to run" iptables set to run

So I either need to poke a hole in the firewall, or let CFEngine manage this for me (later)! To get the Ubuntu host up and running quickly, I simply turned off iptables:

# /etc/init.d/iptables stop iptables: Setting chains to policy ACCEPT: filter [ OK ] iptables: Flushing firewall rules: [ OK ] iptables: Unloading modules: [ OK ] # chkconfig iptables off

Then, because I've been burned by SELinux multiple times, I changed that from enforcing to permissive:

# sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: enforcing Mode from config file: enforcing Policy version: 24 Policy from config file: targeted # edit /etc/selinux/config (change "SELINUX=enforcing" to: SELINUX=permissive

Now reboot the policy server, check iptables and sestatus to make sure they're downgraded.

# shutdown -r now (reboot...) # chkconfig iptables || echo "iptables NOT set to run" iptables NOT set to run # /etc/init.d/iptables status iptables: Firewall is not running. # sestatus SELinux status: enabled SELinuxfs mount: /selinux Current mode: permissive Mode from config file: permissive Policy version: 24 Policy from config file: targeted

And finally, bootstrap the Ubuntu host to the policy server:

/var/cfengine/bin/cf-agent -KI --bootstrap my.policy.server (LOTS of output here...)

Hurrah, my second CFEngine system is online!

Next steps will include:

Make sure my git-daemon is running (via CFEngine).

Setting up a CFEngine policy to poke a hole in iptables to allow policy updates, then resuming iptables (also via CFEngine).

Checking SELinux logs to see if it would be blocking anything important.

Hopefully start writing up my branch naming, testing, and merging procedures for policy files.

References for iptables and SELinux:

https://docs.cfengine.com/latest/guide-installation-and-configuration-general-installation-installation-community.html

tl;dr: Implementing CFEngine policy server, storing policy in a local git repo.

Background

If you maintain one or more Unix boxes, you can benefit from a configuration management system. Want to make sure the root password is changed, on all your systems, to the same new password, securely and automatically? Piece of cake! Want to change an option in all of your sshd config files? No problem! Want to update OpenSSL so that it's not vulnerable to HeartBleed? Totally doable!

I've made several attempts over the past few years to implement a configuration management (CM) system on my home network. I know it's a long- (and even medium-) term win once implemented, but I always seem to run up against one issue or another which derails my progress. I think the underlying problem for me is that I want my network to be "perfect" as soon as I begin implementing CM; instead, what I should focus on is converging on a well-managed network. After all, today's best practice might very well become tomorrow's flawed implementation, so "perfection" is a moving target.

When I first started looking at Open-source CM systems (10 years ago?), I chose CFEngine for the following reasons:

It's mature -- CFEngine has been around for decades(!). It's not quite as old as Unix itself, but it's definitely in the tried-and-true range.

Few dependencies -- You don't need to install a bunch of other software in order to get CFEngine up and running.

Good theoretical underpinning -- Promise Theory underlies everything in CFEngine. All the systems taking part in a configuration are cooperating (rather than coerced) to achieve a desired state, or at least move toward the desired state.

Pre-installation

Here's the (short) list of prerequisites I have before installing CFEngine on my network.

I'll be using an old desktop for my policy server, (Pentium 4 @ 2.8 GHz with 2 GiB RAM). I've set up my DHCP server to give this box a static IP address. There is (was?) a bug in CFEngine that would cause the policy server's IP address to be effectively hard-coded into the network configuration upon the first connection. (My Google-fu is not strong, not finding a reference to it right now.) So I want to give my policy server a well-known, "permanent" IP address.

I installed CentOS version 6 on this system, because it doesn't have a 64-bit processor. (CentOS 7 doesn't seem to play nice with an old 32-bit CPU.) Probably any Linux, BSD, or other Unix variant would work for this. I chose CentOS to gain some more experience with it, and add some heterogeniety to my network.

During installation, I created a user account and added it to sudoers.

After this, just make sure that wget is available:

$ which wget

You should see output like this:

/usr/bin/wget

Installation

Okay, ready to rock! Here's the 1-line command to install CFEngine:

$ wget -O- http://cfengine.package-repos.s3.amazonaws.com/quickinstall/quick-install-cfengine-community.sh | sudo bash

That should get everything installed. You'll probably see a message: Complete! Ready to bootstrap... DON'T DO IT YET!

(Here is a more detailed CFEngine installation tutorial, if needed).

Local git repo setup

I want to have my policy files under revision control, so that I can keep track of changes, undo things that break my network, and hopefully test changes before they cause breakage! I'm using git just to get more experience with it. I'll sort of follow the CFEngine VCS guide. But they use github as a repository server--I don't have a paid github account, and I don't want my policy files to be world-readable, so I'll set up a private git repository on my network.

Git isn't installed yet, and I don't have CFEngine running to do that for me, so I'll install it manually:

sudo yum install git

I set up git-daemon (git repository server) on a separate box, without SELinux enabled (currently). The git server offers read-only access -- I'll use ssh to control write access to the server/repo. The git server is running under my non-privileged account on that box, which means I can manipulate the repository and edit the policy files therein, without having to su/sudo constantly.

ssh [email protected] $ mkdir -p $HOME/git_depot/cfe-home-master.git $ cd $HOME/git_depot/cfe-home-master.git $ git init --bare

The trailing '.git' in the directory name is conventionally used to indicate a "bare" repository, i.e. it doesn't contain a working copy. Shared repositories are normally bare, regardless of how they're served/shared: nfs, git-daemon, file server, http, etc. Note that the filename doesn't turn this into a bare repo, it's merely a naming convention.

Now let's turn the freshly-installed out-of-the-box policy files into a git repo, then push that up to the new repository:

(on the new policy server) $ sudo bash # cd /var/cfengine/masterfiles/ # git init # echo cf_promises_release_id >>.gitignore # echo cf_promises_validated >>.gitignore # git remote add origin ssh://[email protected]/home/me/git_depot/cfe-home-master.git/ # git push -ff origin master

Okay, at this point there are two repositories: The one on the policy server (which has a working directory, but no origin), and the remote/bare repo (which I want to become the "authoritative" one). See Version Control with Git, 2009, pp. 187-8, "Make Your Own origin Remote", for background.

(back on the git server...) $ pwd /home/me/git_depot/cfe-home-master.git $ touch git-daemon-export-ok $ git daemon --reuseaddr --verbose --base-path=$HOME/git_depot -- $HOME/git_depot/cfe-home-master.git

Yes, I am running git-daemon interactively. No, this is not perfect -- see the Background section. But it'll do for now, and I can fix it (with CFEngine!) later. First, let's test the git-daemon. On the policy server, do an "ls-remote":

# git ls-remote git://my.git.server/cfe-home-master.git/ b0b723fa8c254f5a923c101a271a38383e4c5cf4 HEAD b0b723fa8c254f5a923c101a271a38383e4c5cf4 refs/heads/master

Works! You should also see some output on the interactive git-daemon server.

Now let's point the masterfiles/ directory on the policy server to the new git-daemon. We just need to update the policy server's remote:

# edit .git/config ### (remove the '[remote "origin"]' section, including 'url=...' and 'fetch=...' lines) # git remote add origin git://my.git.server/cfe-home-master.git/ ### test! # git remote update # git branch -a * master remotes/origin/master

Success! Moving on...

Bootstrapping the policy server

Remember the Complete! Ready to bootstrap... message from earlier? Time to get that done! We're still root on the policy server:

# /var/cfengine/bin/cf-agent --bootstrap my.policy.server

You'll see a bit of output, ending with:

Bootstrap to '(ip.address)' completed successfully!

Congratulations! Your policy server is up and running, and you're now well on your way to a more perfect network.

Review, next steps

This post is getting pretty long, but I've still got plenty to do. Take a quick glance at this post and you'll notice that the bulk of it has to do with setting up the git server. If a Version Control System (VCS) server was already available (as my dhcp server was), this would have been considerably shorter! Getting CFEngine installed and bootstrapped is really only 2 commands. So if you're willing to hold off on version control for your policy files, you can get this going pretty quickly and easily.

Next step will be adding another system to my configuration. Once you have a real network, (more than just one box), adding more systems is just a matter of "more of the same".

This title is misleading, since I did more than this. But whatever, it's part of a series, I guess.

First, the things that have nothing to do with bad variable exapnsions, in increasing relevancy.

twitter-trollers

I had a shower thought, and decided to create a quicky Twitter bot that would watch a stream for polls, and vote on random results. I do this manually for fun anyway, so I thought it'd be trivial to automate.

Part way through the project, I discovered that Twitter still hasn't exposed a poll API. I'm not wholly surprised, since Twitter has only become more developer-hostile ever since the whole API key revocation scandal years ago.

Code is here.

cookiecutter-pypackage

I discovered a cute little project that can generate language-agnostic project templates. I hate doing scaffolding, so I latched onto this immediately. I forked the author's pypackage template, and stripped out half of the shit I didn't care about, and amended it for my particular python requirements (mypy, only 3.5+).

I'm thinking of putting together a basic C template for some of my simple C projects as well.

My python template is here.

CFEngine PR update

I made fixups to my silent-returnszero patch.

CFEngine testall --verbose

I finally discovered the testall script in CFEngine's tests/acceptance folder. I'll need to amend HACKING.md to include some more detailed information about testall, since it's easy to miss. It's also a pain to develop tests without it.

PR is here.

And finally...

CFEngine now fails promises when variables don't fully expand

It looks like I've almost finished this patch. To be honest, I'm not happy with the implementation just yet. I decided to tackle part of this problem by amending the Rval struct to include a field indicating whether or not that Rval has been fully expanded. That doesn't seem like the best place to put that information, but as an experiment it was the easiest.

After doing that, I was able to run almost the entire test suite without error. This is huge progress, especially since this safeguard will prevent all kinds of CFEngine policy errors.

Commit is here.

Lastly...perl 6

Last time

Today, I started taking a stab at actually implementing the logic to start bailing on bad variable expansions (this is not the finished commit message).

I started right off the bat, by taking all of the extant calls to ExpandScalar and starting to guard their return values, since literally no invocation of ExpandScalar checks the success of variable expansion. However, starting off by guarding these expansions didn't actually work - largely because even on unexpanded variables, ExpandScalar was returning success.

It shouldn't surprise anyone that a function whose return value was never checked returned bogus results in some cases.

After fixing that bug (and running all the tests), the guards suddenly work, and I got this delightful message from my test-case:

error: Could not expand variable '$(bar)'.

My super simple test-case:

bundle agent main { files: "/foo" copy_from => cp("$(bar)"); } body copy_from cp(source) { source => "$(source)"; }

This was already a massive success. I continued guarding, running tests, and I got the commit linked above, with no test failures. I haven't finished just yet, but it's a great start.

One problem I will have soon though, will be guarding variable expansions in rvalues in variable assignments. Currently, ExpandPrivateRval doesn't seem to have any mechanism for reporting error or referencing the calling promise, so it makes it difficult to report error on bad vars there.

A massive problem that I've had at work, is that we sometimes lay down templates, or run commands, or name files after unexpanded CFEngine variables. Currently, CFEngine doesn't have any built-in defense for handling this, and we've built our own series of hacks to deal with this - in most cases.

Today I started work on a patch to CFEngine that will detect this probablem and fail to keep the promise if this happens. After some discussion on the mailing list, we decided that this feature needs to be opt-in for now, and possibly add per-promise options to toggle it. While I'm normally against such dials, this drastic of a change in behavior would break too many installations to turn it on all at once.

Since I had a relatively busy evening, I only implemented the global option, and haven't added any support for failing in this case yet. I'll hopefully get to that later this week.

I had a very quiet and uneventful break from work the last couple of days, and I spent it doing pretty much nothing technical. It was actually surprisingly relaxing - but I'm back on the horse now.

I spent the last hour or so doing some veryh simple work. On CFEngine, I updated my PR to silence the returnszero function. returnszero is a function that executes a command, and returns a boolean indicating the success of that command (where 0 == success). It's unforutnately noisy, in that the command is run using the parent's stdout and stderr, polluting the output of cf-agent. While my patch had been inflight for awhile, we determined that it would be a better idea to expose this output on higher verbosity levels (verbose and info). Honestly, that makes more sense than my original proposal, so I quickly updated my PR.

The other night, I dreamt about a kind of an interesting idea. It'd be super convenient to build a tool that used configuration management to continuously produce container images. In this case, organizations that maintain config management repos for non-containerized systems can use their existing codebase and knowledge to produce new images, and to continuously generate those base images as policy changes over time.

This started as a kind of joke project, until my roommate mentioned that this wasn't actually a bad idea.

I started working on a PoC yesterday, using CFEngine and Docker. I didn't get super far, since we had company over that evening, but I did make a surprising amount of progress. Since I need to call into CFEngine's libpromises, I ended up setting up a lot of the same scaffolding that I've used in my rust kernel modules, which I really need to look at automating. Once I finish my PoC, I'll turn it into something that's actually...good. :)

I also was somehow at a loss as to the best way to create tempfiles in Rust. There didn't seem to be anything in the stdlib to do so, and I couldn't find anything in Cargo. This, obviously, led me to quickly write up some quickie bindings to mkstemp(3) and then publish it on crates.io. If there's a better way to do this rather than writing my own crate, let me know.

Both projects, while in their infancy, are great examples of how trivial interop with C is in Rust. It's been an amazing language to work with so far.