#buildsystem

シェルは最強のDSLだってばっちゃが言ってた / “chefを捨ててシェルスクリプトにした | Ore no homepage” http://t.co/PqCDt82kUh

これと↓のやつについて。

https://twitter.com/sonots/status/591487603028819968 DSL に DSL を被せたがる皆さん

ほんとそうで、Chef にしろ Puppet にしろ Ansible にしろ、シェルコマンドベースでサーバの構築やるとしたら、DSL に DSL かぶせてるようなものですね(Ansible は YAML ですが、まあ YAML + Ansible でだいたい DSL みたいなもんでしょう)。

自分が最近やろうとしてるこれとかも一例と言えそう。

すべての"作業手順書"をDSL化してSCMに放り込みたい

なんで DSL にするんでしょうね？

たぶん、シェルに足りない機能があるんでしょうね。モジュール化とか、関数のネームスペース分けられないとか、…。

あと、シェルだけで巨大なプログラムを作るのは、なんとなく怖い気がします。構成管理ツールにしろビルドツールにしろ、ちゃんとしたやつ使ってればだいたい動くものを、シェルにするといろいろケアしないといけないところが増えそう。シェルでそういう「いろいろ」をケアできるエコシステムがあればいいのかもしれませんけど。

I wrote this buildcloth program that provides a simple python-centric build system tool. The idea is you can write build system code in Python and avoid having to wrangle external build systems that have too much domain specificity or general awfulness.

I plan to use Buildcloth in a few projects eventually, but in the mean time the I needed some way to manage a frighteningly large number of tasks in our build system. So I wrote this function, called runner() that takes an iterable of dictionaries that define a single job, and executes each function (as needed) in a multiprocessing worker pool.

Initially I intended this to be a transitional crutch to using buildcloth, but the truth is that it works really quite well. Sometimes you have to write a few thousand lines of code to figure out what the correct 26 lines of code are. I guess. Here it is, discussion to follow.

jobs must be an iterable that returns dictionaries in the following form:

{ 'target': , 'dependency': , 'job': , 'args': <args|kwds> }

Typically this is a generator function, but you can use any kind of data source.

target and dependency information is only used to determine if a rebuild is necessary. (i.e. if dependency is newer than target) runner() will not order tasks based on the dependency graph. (That's what buildcloth's for.) You can disable dependency checking and just rebuild everything by setting force to True.

The included check_dependency() function can handle lists of targets and dependencies, if needed.

pool designates the size of the worker pool, which defaults to the number of logical processor cores. This is often a good number. A pool size of one is the same as setting parallel to false, which just runs each pool serially without using the worker pool.

Running things serially may yield better results for some classes of jobs, and is useful for testing.

By default runner() returns a count of the jobs run, so you can tell how many jobs you sent were actually run following dependency checking. If you specify results to retvalue, the function will return a generator that will holds the return values of each function run. There are a few other options here which you should be able to sus out yourselves if you care.