The Coding DEAD

Mitchell Hashimoto, creator of Vagrant, speaks about Nomad and Otto http://www.meetup.com/San-Francisco-DevOps/events/227067291/

Nomad  https://ottoproject.io/

Only scheduler support Multi-Datacenter & Multi-Region

Only scheduler everything build-in to one binary 　  it looks like nomad binary has serf & consul.

Only scheduler who do “Optimistically concurrent”　https://en.wikipedia.org/wiki/Optimistic_concurrency_control

evaluation -> allocation -> job “Scheduler” = func(Eval) => []AllocUpdates

Case Study with digital ocean

3 Servers in NYC3

100 Clients 

1000 Job

Addition on v3 in early next year

Log Streaming

Periodic Jobs

Latency Based Scheduling

Driver Plugins

Otto https://www.ottoproject.io/

The successor of Vagrant

Tried vagrant on production but it doesn’t fit in production

Vagrant won’t fit microservices

A wonderfully simple development experience

Codification (Otto) vs Fossilization (Vagrant)

service discovery by consul even if it is in dev mode

otto dev mode runs a VM and uses docker inside it.

I'm trying to find nice solution for supporting MACRO for my go project. (Gohan https://github.com/cloudwan/gohan)

Most of interpreger project in go looks like still in experimental, but Otto or v8 worker are candidates for me.

Otto: https://github.com/robertkrimen/otto v8 Worker: https://github.com/ry/v8worker

Both of project is providing js interpriter for Go. Otto is pure go implementation. v8 worker uses v8 js engine.

I tried simple benchmark. ( increment number 8*10000000 times using goroutine)

Here is result.

$ GOMAXPROCS=1 go run benchmark.go otto parallel test otto parallel thread : 3m30.083406243s v8 parallel test v8 parallel thread : 112.799315ms golang parallel test golang parallel thread : 27.184845ms $ GOMAXPROCS=8 go run benchmark.go otto parallel test otto parallel thread : 1m18.461300405s v8 parallel test v8 parallel thread : 59.017662ms golang parallel test golang parallel thread : 7.621114ms

benchmark code

https://github.com/nati/fun/tree/master/go_javascript

Recentry, v8worker supports sync call, so may be it is good timing to add v8worker support for Gohan project.

[ANN] Gohan project

https://github.com/cloudwan/gohan

http://underscorejs.org/#isString

_.isString("moe"); => true

This is another my fun programming on weekend. I’ve implemented Conway's Game of Life using Google’s GXUI lib.

https://github.com/nati/fun/tree/master/lifegame

The Game of Life, also known simply as Life, is a cellular automaton devised by the British mathematician John Horton Conway in 1970

https://en.wikipedia.org/wiki/Conway%27s_Game_of_Life

GXUI is an experimental Go cross platform UI library.

https://github.com/google/gxui

This toy code can read RLE format which Goly can export.

http://golly.sourceforge.net/

Rotating 3D obj file on the Console! :)

You can display .obj file console using this simple toy code.

You can find the code here.

(1) db struct which is returned by sql.Open() have connection pooling

my bad.. I was creating db struct for each API request.. but actually, db struct have connection pooling, so there is no need to create it per API request

(2) sql.Open don’t open connection actually..

I was checking err from sql.Open, and retrying this if err is not null... However, what I should do was checking return value of db.Ping()

db, err := sql.Open("mysql", "user:password@/dbname") if err != nil {    panic(err.Error()) // Just for example purpose. You should use proper error handling instead of panic } defer db.Close() // Open doesn't open a connection. Validate DSN data: err = db.Ping() if err != nil {    panic(err.Error()) // proper error handling instead of panic in your app }

Some strings get interpreted as Boolean in YAML [1] 

y|Y|yes|Yes|YES|n|N|no|No|NO |true|True|TRUE|false|False|FALSE |on|On|ON|off|Off|OFF

we can use Boolean in Ansible YAML [2]

---

create_key: yes needs_agent: no knows_oop: True likes_emacs: TRUE uses_cvs: false

This is sample playbool.yaml

- hosts: test-servers  vars:      booleans:          - true          - false          - True          - False          - TRUE          - FALSE          - yes          - no          - Yes          - No          - YES          - NO          - on          - off          - On          - Off          - ON          - OFF  tasks:      - debug: msg="{{ booleans }}"

PLAY [test-servers] ***********************************************************

GATHERING FACTS *************************************************************** ok: [localhost]

TASK: [debug msg="{{ booleans }}"] ******************************************** ok: [localhost] => {    "msg": "[True, False, True, False, True, False, True, False, True, False, True, False, True, False, True, False, True, False]" }

PLAY RECAP ******************************************************************** localhost                  : ok=2    changed=0    unreachable=0    failed=0

When I know puppet first time, I got very excited.  Puppet helps me a lot for managing 30 machines. However, I found it has also pain. It was hard to debug because execution order is not as written.  There was also a lot of pain of recipe dependencies. I tried some 3rd party recipes but most of it won’t work or don’t suit my use case.

Chef boomed after puppet. Cookbook on chef was just a Ruby DSL. Even if I don’t know how to write ruby, DSL was easy to understand [Note1]. It was easier to debug, and dependency management was not so pain as puppet. However, I faced chef server node down issue because of CouchDB created ton’s of cache files. (Note that this is old issue, and I believe they don’t use CouchDB backend anymore.) Anyway, there were burden of provisioning master server management.  We should operate it & manage it. (HA, backup etc.)

Fabric got popular after those days around me. I’ve using python for most of work, so Frabric suits me. Server-less architecture looks nice for me. When the fabric script grew up, there was pain of management python “Code”. We should be responsible for write nice python provision code, which are extensible & fault torrent.  

I thought “OK, provisioning is naturally painful.. Anyway, we should fight with many Frankenstein like middle ware & application servers and configurations.. Provisioning tool doesn't matter..”

Now, I’m using Ansible. Ansible looks like YAML DSL + Fabric.  YAML DSL works nice to hide burden of manage “Code”. Debugging is straightforward.  Actually, We can run from middle of provisioning using --start-at-task parameter. This feature makes debugging faster. Sever-less architecture. Clean configuration variable management.

I was testing ansible + keepalived + vagrant using this repo.

https://github.com/tcomerma/ansible-keepalived

Example on Readme works fine if I setup correct ip on Vagrantfile

mkdir -p test-keepalived/roles cd test-keepalived/roles git clone https://github.com/tcomerma/ansible-keepalived.git cd .. cp roles/ansible-keepalived/test/* . mv Vagrantfile.Debian Vagrantfile vagrant up

I’ve used vagrant halt & vagrant up to test failover and failback.. However failback didn’t work. This is because vagrant clears all exisitng ip aliases on start up after keepalived process comes up.

I need small & easy-to-run partial integraion test env, so I’ve decided to use go test module for small integraion test with env variables & skip test api.

[Backgorund] I’m writing API server which can be connected to SQLLite3 & MYSQL using golang. I was using SQLLite3 backend for unit test. However, SQLLite3 and mysql are diffrent beast.. so I get start suffering small difference of two diffrent db backends. Of couse, I have vagrant-based integraion test enviroment, but it takes time to setup everything including big backend servers. I need light-weight integraion test env, which can be used something like UT. Actulaly, this kind of small integraion test is really nice to find bugs in early phase.

so I decied to write integaraion test for mysql using same way with unit test using go test modules.

[Solutions]

I found three way to filter test in go test. (1) -run regexp (2) build tag (3) using env variable & skip api.

(1) -run regexp

go test takes -run parameter, so that you can filter test regexp.

-run regexp    Run only those tests and examples matching the regular    expression.

I didn’t choose this one because I don’t want to introduce additional naming convention for test names.

(2) build tag

we can put “build tag” for go source code

such as

// +build linux darwin

see http://golang.org/pkg/go/build/#hdr-Build_Constraints

go test takes this tags.

go test -tags=integraion

I didn’t choose this one because we should speparate whole test file for integraion test, and it sounds like not so handy. I would like to put both of sqlite3 & mysql test code in same db_test file because they shares most of test codes.

(3) using env variable + skip

Finally, I found nice way which fits me. This is sample test. so we can add mulitple env switches on test. (also we may put connection parameters in the test also)

func TestSomeProtectedResource(t *testing.T) {  if os.Getenv("SOME_ACCESS_TOKEN") == "" {    t.Skip("skipping test; $SOME_ACCESS_TOKEN not set")  }  // ... the actual test }

see https://splice.com/blog/lesser-known-features-go-test/

[Conclusion]