Hack.FoolAround.Repeat

Recently , I wanted to setup an easy and quick monitoring dashboard for a server. I just wanted CPU , RAM , Network metrics and maybe the running processes. Much like what I would get from htop . I found ScoutRealtime which provides exactly that . [https://scoutapp.github.io/scout_realtime/]

Here is the code to install it. Just make sure that port 5555 is open on your server. Note: The script installs Ruby 2 and switches to it. Customize according to your needs

I have recently noticed that there are a few functional tests that run for more than the acceptable time and/or hang . I decided to write a quick Bash script , and make it a cronjob , to find the browser [firefox] instances that run for more than 10 minutes [can be changed anytime] and kill them [brutal, I know!] .

If you read the CD documentation you will notice that CD ,when connected with GitHub for automatic and continuous deployments on every single PUSH, will deploy always the default branch of the repo that we connected it with.

Yeah but that sucks! No developer ever didnt used branches! [or did he/she?]

So what is the solution to this otherwise flawlessly automatic deployment pipeline?

After crushing and banging and burning somewhere along StackOverflow, AWS CodeDeploy forums and personal blogs ... it came to me. I will change the default branch of the repo, if that is what CD plays along with. :]

So I wrote a snippet [using the GitHub API] doing just that. [I could do it easily from GitHub UI, but hell! Where is the fun and repeatability in that???]

When I made the switch from my Economics studies to All-Things-Programming(TM) I decided to write down every single [small or big] project that I would work on and could prove my knowledge and skillset. I wanted to do that in order to remember what I have worked on through the years and because I strongly dislike vague words in resumes that were put there to dazzle by their lack of detail. What better way of someone getting to know what you know by telling him\her [If he\she has the time to listen].  Let’s say I wanted to create a log of my evolving skillset. I am by nature curious and always want to learn more.  That started with a Google doc that quickly got transferred to my LinkedIn profile. But I found out soon enough that even there , the characters are numbered. 

So, I decided to keep that log in my [b]log . See what I did there? :P

So , here it is .... [Note: It will  always be under editing :] ]

Object­ oriented test design using open source tools (Selenium WebDriver/RemoteDriver, unittest,nose, py.test,splinter etc. ) 

Working in an environment of rapid development with Agile principles .

Setup of all the QA workflow and infrastructure(TestRail , Selenium , VPSs , Jenkins , MySQL DBs , Grid , Test Parallelization, GitHub webhooks)

Selenium Grid implementation with several virtual machines (Win7/XP,Xubuntu,Hackintosh Mavericks) . Parallelization of test runs . Management of the grid with Vagrant/Python to open/close the VM s on demand when the tests run.

Selenium Grid implementation on private Cloud (private Hub/Nodes boxes).

Selenium Grid implementation on Amazon Web Services's AMIs (on demand auto spawning of instances).

Automation of test-reporting procedure to TestRail with it's API.

Load testing of web applications using Locust , Bees with Machine Guns, Selenium Grid. Development of realistic HTTP load tests and detailed analysis of the produced results.

Application scalability analysis and bottleneck identification based on AWS CloudWatch metrics.

Implementation of SMS-alerts via the Twilio API for immediate warning of the QA team on critical failures. [That was more for fun and specific team members annoyance than real world use :] ]

Selenium Webdriver (IEdriver , Firefox , Chrome , PhantomJS headless browser).

Provisioning and administration of Linux boxes (Ubuntu / Xubuntu / Mint).

DevOps operations (optimizing NginX , Bash scripting for console automations , Migration of paravirtual AWS Linux to HVM , Jenkins customizing / API implementation , Git/GitHub integration).

DevOps automation and Continuous Integration/Deployment with Fabric, Ansible, Docker

Development experience with Flask, Jinja

Implementation of ELK [ElasticSearch / Logstash / Kibana] stack for centralized log monitoring of AWS main / staging / testing infrastructure using Docker containers 

Setting up continuous code deployment from IDE to server using Jenkins and GitHub or BitBucket

Management/Orchestration/Automation/Provisioning of AWS resources [instances, security groups, keys etc] using Ansible and YAML .

Creation of a dockerised Selenium Grid infrastructure with reporting capabilities [using Testrail] to parallelize the testing of the developers branches

Modelling of the entire company infrastructure using AWS CloudFormation as a part of the Disaster Recovery strategy

Set up of the company’s passive ‘Pilot Light’ infrastructure as part of the DR strategy

As I am working lately with Docker in a side project, I stumbled upon the need to create a host volume directory [inside another general directory] for every new user of a service with an arbitrary prefix.

As I am also using Ansible for the same project , I decided to make a playbook to handle this.

#Playbook to create a new directory in a specific path #Use it like this: #ansible-playbook -i hosts --extra-vars '{"PREFIX":"something"}' create_new_client_volume.yml --- - hosts: localhost  connection: local  vars:    prefix : "{{ PREFIX }}"  tasks:  - name: Create new dir    file: path=~/some_path/"{{ prefix }}_volume" state=directory

I was trying to start a Locust master server and register it’s slaves to it. So I needed to grab the internal IP of the master server dynamically [cause all instances are inside AWS]. 

So here it is. Use it as a variable and you are good to go.

Working on a personal project (see other posts) I found myself in need of automatically setting up a Jenkins [docker] container. The need was to set up a new Jenkins job [with user provided data] after provisioning the container. 

After a long and hard safari for answers on StackOverflow,Google etc and a long fight with the terrible Jenkins XML [god make it] API , I ended up with a cURL command that creates a new Jenkins job.

curl -X POST -H "Content-Type:application/xml" -d "<project><builders/><publishers/><buildWrappers/></project>" "http://USER:PASSWORD@JENKINS_HOST/createItem?name=NEW_JOB_NAME"

For more and more time lately I enrich my QA pipelines and practices with DevOps. That includes automation of the build pipeline, continuous deployment of code in a fleet of instances on AWS, provisioning instances and containers and many more. One of my latest discovery that I want to share is envassert . Basically is a Python implementation of serverspec (Rspec tests for server configuration) and uses Fabric under the hood. (That kindly reminds me that I want to restart learning Ruby and Rspec...:] ) 

Many times at work I have reported bugs that had to do with a production server that someone mingled with and broke it’s standard configuration. 

Lately I started using Ansible. And what a tool it is really. Having worked quite extensively with Boto for various things , continuous deployment among them , and Fabric , I must say that Ansible is a fresh breeze and a time saver. I had started to build a whole library with boto functions to do  various tasks that with Ansible is just a YAML file.

Well , I want to share the candy so I started a public repo on GitHUb which I hope to enrich with a lot of playbooks.

Ever wondered “How can I make sure on a Linux terminal if a service is running?” ? Type ps -A and hit enter . There they are all of them! (or...something like this ps -A | awk '/[process-name]/{print "I am up";print “I am down”}' )

Note to myself ... and anyone not willing to lose 2 days scavenging the internet for a solution.

When you try to run a python script (like a fabfile) from the Windows terminal and it refuses to run by spitting out an ImportError: module not found it means that the path of the project’s folder is not in the sys.path for Python to find it.

You have 2 choices.: The temporary one: Add the path in the sys.path on the fly (on script execution, and per script) like  import sys,pprint sys.path.append("C:\Folder\Python27\Projects\folder\path") # pprint.pprint(sys.path) #it prints all the paths that python searches for modules

Due to new requirements at work I can tinker again with my lately favourite Boto python library for AWS. I need to keep our staging DB rationally up to date with the live. The way we decided to do it is by replicating the live DB (make a read replica) , make it RW, attach it to Staging server and delete the (now) outdated staging DB. 1. At first, let’s make a read replica of a specific RDS db instance.

To be continued..

Some months ago , at work , we were on a hunt for a nice to learn and play with load testing tool.

It had to be open source. I needed to know that I can dig deeper in it and hack and slash as desired. I reviewed any such tool that I could lay my hands on. Finally I came to the understanding that Java is the go-to language for load testing. Obviously I stumbled upon JMeter, Grinder and Gatling. 

But then I asked myself. Why Java and not Python? Disclaimer: I don’t know Java and some times that I tried to pick it up I found it to be irrationally unfriendly. On the other hand , I write Python some years now. It is childish to learn and really powerful. It is also the (black,white,grey,pink...) hackers preferred language. So why is not Python the go-to lang for a hacker-friendly scenario of hammering a web app??? Of course I tested Funkload (hated it...sorry.Yet another wasted day!), Multi-Mechanize (could not make it’s dependencies work on a WIN7 machine and gave up under time pressure..) and Pylot (cant remember why I didnt finally used it but I bet would be a fine reason). These are the only Python related load testing tools in this list by the way... I decided to give a shot to Locust, which I found without it being mentioned anywhere. It is the work of Jonatan Heyman and Ronnie Kolehmainen at ESN. It was built to test Battlefield’s Battlelog.

And it is actually really nice. It is very powerful and with a little customization did the job that others couldn’t (maybe were too complex) . 

So , I liked Locust so much that I envisioned a SaaS around it. I would love to dig deeper in it and combine it with the power of AWS (I already have cooked some scripts hehe ) , build a front/back end with Flask (Locust is built with Flask and I want to finally master it, oh what a coincidence) and add something like Grafana/Graphite .  

Yeah , it would be a really great tool.

Smells like startup ha...  So let’s get down to business ! :]

EDIT_1

I decided that I need to learn more about Fabric .  It is a Python library that ... well essentially it’s a python wrapper of the Linux terminal. It does with python anything that the terminal can. Remotely. How cool is that! I think it will immensely help the automated operations on multiple AWS instances . EDIT_1.1 For now , I may stick with Fabric even though I already stumbled upon a limitation it has (cannot send a process to the background and detach the shell natively) but eventually I will give a go to Ansible which claims to do the same job on a higher level and better.

EDIT_2

Fabric is awesome but it makes my life difficult cause it does not provide a way to detach the shell from the process I initiate (I start a locust server to an instance and I want to continue with the other 1435 instances and I cannot..)

Today I decided to cool off my brain cells from Fabric and Boto and setup a web development virtual environment for Flask. It’s time for the project to take an initial shape ... and not just be scripts and functions :] I created a venv on PyCharm and installed all the modules that I think will be useful. (some_name) C:\Users\wehappyfew\PycharmProjects\ some_name>pip install flask requests boto fabric locustio pyOpenSSL mitmproxy python-jenkins cookiecutter

EDIT_3

Warning : Too much reading ahead! :] :]

A snake runs underneath.... :]

EDIT_4

In the picture above I didn’t include Docker. What a nice thing it is! I have dove in Docker all weekend and I already found two case studies . One for the current project and one for work.

For this project:  I will attempt each user that wants to run a load test to work inside his very own container for his master server (I was thinking each one to have his own micro instance at first..).The master server ,from experience is very light on resources and only needs an open port. TODO->Make a post about it!

For work: I want to build an image with the application’s master branch (which should always be up to date), and every developer will build a container with his branch on it. That way would be several branches on staging ready for testing (no more ‘My branch should go first!’ kind of fights). But that way we will surely need more testers! :O Hehe, I create jobs! TODO->Make a post about it!

EDIT_5

I haven’t write for a couple of weeks . I must admit that I made significant progress with the backend , cause it felt more sexy and tempting than fiddling with Bootstrap.  I have made a mockup of the needed pipeline [after countless iterations] in Draw.io . TODO - Post it here - I decided ...for now... to go with 2 docker container for each user. One for jenkins to fetch and build the code and one for the Locust server. They will both share a host volume containing the code that Jenkins fetched. I have also made an Ansible Playbook to start AWS instances and create key and security group to communicate with them.  Additionally , I wrote a cURL command to dynamically create a new job to every jenkins container and set the user’s repo url, git url and branch name [will be entered from a UI ].  Major complain: I would expect that Jenkins as an industry standard in CI , would be a breeze to use programmatically ... but boy was I wrong as HELL ! It is a pain in the ass to manipulate it. cURL command from shell to create a new job???? OMG and 3 LoLs ! Anyway, the journey until now is educative and rewarding as hell. I play and dive deep head first to cutting edge techs , reading tech blogs of Fortune 500 engineering teams and learning a ton of stuff.

EDIT_6

During load-testing with Locustio , I found myself having spawned 10 Locust slaves on AWS. 

Unfortunately, Locustio does not have a web interface (or something equivalent) to centrally manage it’s slaves.  And the thing with the slaves is that each and every one of them must have the latest and updated code that is going to run on the master. How is that possible? Oh well, with Jenkins CI installed in each slave. So, I had to build the same code in 10 different AWS instances. The first and immediately most productive solution is open 10 Chrome/Firefox tabs and go to the Jenkins urls. But as you can imagine any good engineer that gets bored easily ... well,is bored of the amount of clicks included.

So , I sat down and crafted a small piece of code to build all of them at once. As someone can see I didn’t used their IPs, but if someone wants to do it that way is pretty straightforward.