The Turing Experience

There are two different approaches to monitoring your applications. From the outside, in, using a monitoring service like Scout or New Relic. Another way is from the inside out. MySQL has a very convenient tool called the Slow Query Log. 

dev.mysql.com states, “The slow query log consists of SQL statements that took more than long_query_time seconds to execute and required at least min_examined_row_limit rows to be examined.”

You set the variables and then mysql keeps logs on any queries that meet or exceed those parameters.

By default, the slow query logs are disabled. To enable the logs, you need to insert a command into the my.cnf file. On a VPS the path is usually /etc/my.cnf

In the mysql section (labeled [mysqld] add you choice of slow query logging. e.g.

long_query_time=2 #sets (in seconds) what counts as a long query time. (default is 10)

log-slow-queries=/var/log/mysql/log-slow-queries.log #to state where you would like to place the file. If unspecified, The default is (host_name)-slow.log.

you can also monitor unindexed queries or admin statements:

log_queries_not_using_indexes

log_slow_admin_statements

If you are going to monitor all non-indexed queries, you might want to limit the rate that they are put in the logs (as there can often be quite a few). 

log_throttle_not_using_indexes=12 #limits the number of logged non-indexed queries per minute to 12 (or whatever number you want). The default is 0 meaning there is no limit and will record each one. 

Logging non-indexed queries can be useful in order to determine if a table column starts becoming queried often and/or bogs down and should be indexed.

Make sure you create the folder and file for the slow log to write to:

=> touch var/log/mysql/slow-queries.log

IMPORTANT NOTICE: MySQL will not start using these new commands unless you restart it:

=> service mysql restart

wait for some requests to ping your server and query your database and then open up the logfile to check for and slow query logs. 

/root/myapp/var/log/mysql/slow-queries.log, Version: 3.23.54-log, started with:Tcp port: 3306  Unix socket: /tmp/mysql.sock Time                 Id Command    Argument # Time: 032216 15:03:33 # User@Host: ghudson[ghudson] @ localhost.localdomain [127.0.0.1] # Query_time: 13  Lock_time: 0  Rows_sent: 117  Rows_examined: 234 use wsdb; SELECT * FROM users WHERE created_at>'2016-03-10 00:00:00';

Daemons can be very useful (or scary apparently). For a fun read, both Katrina Owen and myself recommend 

In the real world, it is often a good idea to run background workers to perform tasks that would otherwise bog down a server or a user’s experience. A common example is sending off an email when an event happens. Right after the event is triggered, without a background worker, the user would have to wait for the process to complete before continuing on with browsing/usage. 

Sending the task off to a background worker to complete allows the user to continue their activity uninterrupted. 

While it is certainly possible to write your own background workers and it may help with managing dependancies. As Daemon newbies, lets start off with a gem to help for that. 

Enter sidekiq. They tout easy to use and very fast workers using client APIs, server APIs, and Redis. Plugging it in is simple enough. 

gem ‘sidekiq’

create your task you want the worker to perform. Note: create one class for each different worker task you are generating. Include Sidekiq::Worker. Then create the perform method. For sidekiq it is always called perform, so do not alter this. 

class NameOfTask     include Sidekiq::Worker     def perform(args)         # stuff to do     end end

in your controller (or where you are calling the worker task if you are not necessarily using Rails) assign the task. .perform is an instance method, and any of the perform_later options are class methods (NameOfTask.perform_later). 

Sidekiq recommends if your tasks are using lots of database calls, to turn up your pool in your database configuration to accommodate asynchronous activity.  

make sure you have bundled your gem, and then run bundle exec sidekiq.

To start running the workers, it is helpful to have multiple tabs open on your terminal.  

In the first tab,  start your Redis server: redis-server

then the client side in tab 2: redis-cli

and finally in a third tab, start sidekiq: sidekiq

The sidekiq server will pull the jobs from the queue in Redis. It will run the tasks as defined in your perform method. 

Keep in mind that in a real world operation, there might be multiple different tasks from multiple different job classes running concurrently. Keep the tasks simple and make sure they point to a persistent source of information and not something that could be in flux. Limit the responsibility and power of each worker. Keep yourself protected!

Rails comes with caching functionality straight out of the box! Caching can help speed you application load times by retaining data for reuse instead of reloading it from the source. 

But it doesn’t always know when to update. As many (wise) developers have stated, “There are really only 2 tricky things in programming: 1) naming things, 2) Caching strategies, and 3) Off by one errors.”

You can pretty much cache anything, including page elements, database queries, and fetching 3rd party data.

Especially for larger, more complex pages, caching page elements, can help reduce load times as a user navigates through your site (this only works if there are reoccurring elements or if they have to go back to previous pages!)

After a point, it becomes impossible to improve database queries through optimizing the calls. You can cache these so that the information only needs to be grabbed one time. This can be done all at once for smaller queries, or in ‘chunks’ for large amounts of data. 

This works similarly for 3rd party data. 

A general rule of thumb is to cache anything that takes longer than 200ms (or about 20 times as long as it takes a trap-jaw ant to bite - from stimulus to jaw close) to load. 

While some people get cache happy, it’s important to use a tool to find out where your bottlenecks are. Caching probably won’t solve your n+1 queries!

If you want to play around with caching in development, make sure you change your default settings. In your config/environments/development.rb make sure you have

config.action_controller.perform_caching = true

To cache HTML elements using Rails, simply surround what you want to cache in a cache do block:

<% cache do %>

    <div id=“stuff-to-cache”>

        <!-- stuff -->

    </div>

<% end %>

Maybe you want to cache an items list (Item.all). But what if another item was either created or updated, how can we tell the app to dump the cache and load the new data? 

Here you want to create a cache key that is unique based upon the criteria you give it. For instance, if you set the cache key based upon the total number of items in the database and the last time any of them were updated:

cache(”items_count -#{Item.count} -#{Item.maximum(:updated_at)}”) do

    #stuff

end

It will know to dump the cache and update the information if an item is either added, deleted, or updated since the key would change. 

If you are doing a lot of caching, you can even create helper methods for each type of AREL cache action you want to perform.

def cache_key_for(model_class, label = “”)

    prefix = model_class.to_s.downcase.pluralize

    count = model_class.count

    max_updated_at = model_class.maximum(:updated_at)

    [prefix, label, count, max_updated_at].join(”-”)

end

and then you could use your helper method to plug in anything* you wanted to cache from the database.

cache( cache_key_for(Item, “count”)) do

    #stuff

end

*be careful what you cache...

It is also possible to nest caches so that when something changes, only the lowest level that is affected is updated and the rest remains cached. This is aptly named ‘Russian Doll Caching.’

<% cache(”items_count -#{Item.count}”) do %>

    <% @item.each do |item| %>

        <% cache(”item_updated_at -#{item.id}-#{item.updated_at}”) do %>

            <h1><%= item.title %></h1>

            <h2><%= “Cost: #{item.price} Quantity: #{item.quantity}” %></h2>

        <% end %>

    <% end %>

<% end %>

Now when an item is updated, only the cache for that item is dropped and the item is reloaded into the cache from the source. The item id is thrown into the key just in case more than one item shares an updated_at timestamp.

It can obviously get much more complex than this, but I hope you have gained a basic understanding of the idea.

While most of the days were spent applying knowledge, we did run across a few new things. 

When using jQuery’s .each method, it doesn’t scope .class hooks, so it will iterate through each element that matches on the page. 

Manifests are files that tell Rails what to include in the asset pipeline. You may have peeked into your assets/stylesheets/application.css and assets/javascripts/application.js which come with some files preloaded to be used in the asset pipeline by default.

//= require jquery //= require jquery_ujs //= require_tree

You can require specific files,

//= require filename

a specific directory which only uses files in the current directory and not any subfolders.

//= require_directory

or the whole tree starting with the folder that the document calling the require is in as the head.

//= require_tree

You can also use:

//= require_self

To load all the assets in the current file before any dependancies but this is often considered bad form as manifest files should be a directory only, not contain code.

Another rarely seen command in a manifest is:

//= depend_on filename

which announces a dependency, but does not load the file.

//= stub

blacklists a file for the rare instances where you need to have an assets that is not run through the asset pipeline.

Is this you?

I know we already talked about this, but understanding the asset pipeline is pretty important for ensuring your applications deploy as you want them to. 

Practicing with the production environment on your own development machine is a good way to understand how things work and what can go wrong.

You can do this easily in two ways. 

$>RAILS_ENV=“production”

$>rails s

or in short hand,

$>rails s -e production

I tried quite hard to find a difference between the two. I think the RAILS_ENV keeps the state set as production until you set it back where the second option only sets it for the one instance of the server. But if you close that terminal and reopen the environment will revert to the default. 

Precompiling your assets lets you mimic the processes that happen when your application is deployed. 

$>rake assets:precompile 

compresses, concatenates and minifies your assets. While this makes them take up less memory, it isn’t very easy to read anymore. Another thing this does is fingerprint the file. This makes it easy to know if there has been an update to the assets so that new files can be loaded instead of using cached data.

If there are conflicting commands, the command that gets runs last gets precedence. After compiling, if you go into the rails console

$>rails c

you can find out what will be run.

]>Rails.application.config.assets.paths

will give you the load path. Whatever is on top will be run last and wins any same name conflicts. 

If you want, you can append files (such as this example, config/initializers/assets.rb

]>Rails.application.config.assets.paths << Rails.root.join(”config”, “initializers”, “assets”

Then whatever you put into that file will be included in the asset pipeline. 

Wait!  ...no not that AJAX... we need PROGRAMMING talk AJAX

A - Asynchronous

J - JavaScript

A - And

X - XML

Ajax is used to send and receive data in the background (asynchronously) from the client side. This means you can have a page update dynamically as your intended user’s computer makes requests you designate. You get to free up your server from repeated requests from the client AND keep your server from making multiple requests of it own... In short, you are using the client computer to serve itself, taking advantage of relatively fast on average client side computers in order to free up your own servers to operate more efficiently/smoothly/quickly. Due to this, the user experience is usually better as well, since they usually have to wait less time to get and update the data on the page they are viewing.

$.ajax({

   type: ‘GET’,

   url: ‘http://fetch-json-data.com/api/v1/endpoint’,

   success: function(data-you-get-back) {

       //whatever you want to do with the request data you receive

   }

You can make ‘POST’ requests as well so that you can utilize all CRUD functionality through AJAX.

It’s not without its flaws. It violates same origin policy, causing many servers to question the nature of the request (is it malicious?). Unless servers are set up to accept these requests, the requests will be denied. A workaround for this is to send the requests back to your server and have your back-end make the request and then pass that information back to the client, effectively making your server a middle-man. This partially removes one of the strengths of AJAX and reburdens your server with making multiple requests and having more requests come in. The only redeeming part being that the requests are usually for smaller amounts of data than a complete page refresh would be. Also, due to being asynchronous, there can be additional complexity introduced which can lead to bugs that are difficult to find and increased difficulty in crafting good, stable tests.  On top of this, AJAX requires that the client poll the server instead of just listening, increasing the overhead/increasing their latency.

While we get ready to build our personal projects, we are assigned an exercise to help identify and relate to our intended audience. Personas are kind of like a mini-biography of a fictitious person who might use what you are wanting to build. 

To complete a persona, have four sections, labeled Persona, Demographics, Habits, and Fears/Aspirations.

In the Persona section, actually draw out your representation of a user. Here, I have Lydia Chavez, a recently pregnant and divorced mother. 

I also wrote out some things about her

In the demographics section, put personal statistics.

In the habits section put down what this person does regularly.

and then list their fears and aspirations (like a fear of posting a sideways picture!)

Sorry for the water damage!

Your ActiveRecord model objects have an .as_json method. The default takes ALL the table’s attributes to create a Hash that it then calls a more familiar .to_json upon. 

It is often the case that you will want to change this default. Perhaps you only want a few of the model’s table columns represented in your JSON string. As with all methods coming from parent classes, It is possible to overwrite this method.

def as_json(options = {})

    super( except: [:created_at, :updated_at] )

end

The ‘super’ mirrors the method from the parent, then we change it to only do what we want it to (in this case return the model object minus created_at and updated_at).

For more explicit serialization, there’s help (’there’s a gem for that’). gem ‘active_model_serializers’ helps with many aspects of serializing data. After you put it in your gemfile and bundle it, you can run this command to create a file to specify what you want serialized:

 rails g serializer model_object

will create the file with the model object’s table columns as default data to be serialized. If you do not have a model set up for what you want serialized yet, you can use

rails g resource model_object column_name:data_type column_name2:data_type2, etc

If you want to serialize data that isn’t an AR object, it still should be framed in the same structure and pass the same restrictions as set by ActiveModel::Serializer::Lint::Tests.

Set these up as you would a model (if you based it off an already existing model, it will come with the same setup by default). This example is straight from the rails-api github page.

 https://github.com/rails-api/active_model_serializers

class PostSerializer < ActiveModel::Serializer    attributes :title, :body    has_many :comments    has_one :author end

class CommentSerializer < ActiveModel::Serializer    attributes :name, :body    belongs_to :post_id end

With such a setup, when you serialize a Post, it will also serialize all the Comments associated with it by default. 

Your "Interface" is the methods you expose. For your API, it is the methods you expose through JSON, XML, etc. We will be focusing on JavaScript Object Notation for our next project as it is becoming more popular than XML and other API formats. All JSON is valid JavaScript, though no functions are allowed. The Values can be strings, numbers, booleans, arrays, and hashes.  The keys MUST be double quoted strings "{ "key1": value1, "key2": value2 }" and look much like a Ruby Hash. The JSON object is sent over HTTP as a string.

If you have been following my blog, the prior paragraph is review. But, remembering all that, it will be easy to understand what we want to generate and therefore easy to make our expectations for our tests. But first, we want to make sure we routed our endpoints correctly. We will want to be using the responders gem again.

def test_index    get :index, format: :json    assert_equal :ok, response.status #or assert_response :success end

Next, we want to make sure the data is being passed as a JSON object and that the correct number of objects are being preserved in the conversion to and from JSON.

def test_returns_number_of_items    number_of_items = Item.count    get :index, format: :json    json_response = JSON.parse(response.body)    assert_equal number_of_items, json_response.count end

We may also want to test the creation of an object through API:

def test_create    new_item = { name: "item1", description: "very fun" }    post :create, format: :json, item: new_item    assert_response :created    assert_equal "item1", Item.last.name end

The test for update and delete would be much the same.

def test_update    item = Item.create(name: "item1", description: "very fun")    updated_item = { name: "Better item1", description: "even more fun" }

   post :update, format: :json, id: idea.id, idea: updated_idea    assert_equal "Better item1", Idea.last.name    assert_equal "even more fun", Idea.last.description end

 def test_destroy    item = Item.create(title: "idea1", body: "very fun")    starting_count = Item.all.count

   post :destroy, format: :json, id: item.id

For authentication through the API we built, we can still use the bcrypt gem and keep most of the setup the same. 

class User     has_secure_password     ... end

rails g migration AddPasswordDigestTouser password_digest

if you skip over to rails c, you can test the .authenticate method. Create a User and then User.last.authenticate(’correct_password’) #=> true and User.last.authenticate(’incorrect_password’) #=> false

You can set up your controller to authenticate with a user email and password:

Api::V1::WhateverController < Application Controller     before_action :authenticate!     ...     def authenticate!         request = please insert your email and password”         authenticate_or_request_with_http_basic(request) do |email, password|             user = User.find_by(email: email)             if user && user.authenticate(password)                 true             else                 head :unauthorized             end         end end

Or, you can have the user authenticate with a token:

It’s often wise to separately namespace your API endpoints instead of sharing one controller and changing the response based upon the format the request comes in as. 

For instance, if you were to have it all in one items_controller.rb:

def index     @items = Item.all     respond_to do |format|         format.html { @items​ }         format.json { render json: @items​ }     end end

Assuming you have left the defaults alone in your routes, visiting /items still gives the html. Visiting /items.json would render the json. You would get an Unknown Format error if you visited /items.xml since you didn’t specify that it should respond to that in the items#index.  

You could repeat this for each of your endpoints...

def show    @item = Item.find(params[:id])    respond_to do |format|        format.html { @item }        format.json { render json: @item ​}    end end

...or pull the respond_to out of the actions with help from the responders gem:

class ItemsController < ApplicationController     respond_to :html, :json, :xml

    def index         @items = Item.all         respond_with @items      end

end

There. Much nicer. But then we want to have a create action and it starts to get messy again:

def create     @item = Item.new(item_params)     if @item.save         respond_to do |format|             format.html { redirect_to :items_path, notice: “Item Created”             format.json { render json: @item } #responds with the errors         end     else        respond_to do |format|            format.html do                 flash.now[:notice] = @item.errors.full_messages.join(”, “)                 render :new             end            format.json { render json: @item }        end    end end

Wow. That’s a big, messy, action. At some point we might realize that we are interacting with two kinds of entities; humans and computers. It makes sense that we should have two separate controllers. This also would have a nice side effect of cleaning up the actions. 

Lets start with our routes.

Keep the human interface routes the same. For the API routes, make sure to namespace them within an api route AND a version route. Also, new and edit pages are not needed when rendering API data (API requests POST directly to create and update). You can also expect a certain format as your default so that your endpoint would not need .json at the end of the url:

namespace :api do     namespace :v1 do         resources :items, except: [ :new, :edit ], defaults: { format: :json }     end end

the url:

/api/v1/items

would be the same as

/api/v1/items.json

Controller Time! Create another items_controller.rb inside new folders:

/app/controllers/api/v1/

Inside the controller file, you have to show the namespacing as well. I still recommend using the responders gem:

class Api::V1::ItemsController < ApplicationController     respond_to :json, :xml

    def index         respond_with Item.all     end

    def show         respond_with Item.find(params[:id])     end

    def create         respond_with Item.create(item_params)     end

    def update         respond_with Item.update(params[:id], item_params)     end

    def destroy         respond_with Item.destroy(params[:id])     end

    private

You don’t always want to go out and ping each API endpoint your application consumes every time you run your tests. Also, often in a development environment, you may find yourself without internet connection. Tests which depend on API data would fail.

Enter VCR (yes, just like that 1970′s contraption you could record and rewatch shows on)!

Make sure you set up VCR correctly so you don’t get your tape stuck in the machine. 

in gemfile.rb

gem ‘vcr’ gem ‘webmock’ #helps stub http requests for your tests among other uses. 

in test/spec_helper

require ‘vcr’ VCR.configure do |config|     config.cassette_library_dir = ‘test/cassettes’     config.hook_into :webmock end

in your _test.rb file, tell VCR where to start and stop recording using a block.

def test_name     VCR.use_cassette(’testfile_name#method_name’) do         api endpoint consumption setup         assertations     end end 

When you use VCR the first time, it is just like running your tests without VCR except it literally records each endpoint your tests hit. 

It saves these in a cassette folder (or whatever you named it in your configuration). Make sure to .gitignore this. VCR then uses the recorded data if you run your tests again.

If you ever want to reach out to the API endpoints in your tests again, simply delete some or all of your cassettes. 

There is a strong movement toward making any application serving APIs into a RESTful convention. With RESTful applications, you can expect a few things from the API you are expecting to access and consume. First off, it will most likely have a specific URL just for accessing that API. Secondly, you can also expect the API to be versioned so that when a change is made to the API you are consuming, you have some time to adjust your app using the old version instead of your app just breaking each time a change is made. 

RESTful APIs are must easier to digest than non-RESTful, especially if there is limited or no documentation!

For our first attempt at consuming APIs we will be using the Hurley gem.

Lets create a PORO for accessing and parsing the JSON API data we are getting. 

class ApiParser end

Once we have hurley set up, we access the API by first establishing the connection (and saving it to an instance variable). Lets do this when we initialize our PORO

class ApiParser     attr_reader :connection

    def initialize         url = ’base api url of the app you are consuming’         #(something like ‘https://www.github.com/api/v1′)         @connection​ = Hurley::Client.new(url)     end end

We then access specific information we want by sending the ‘get’ method to our connection (kind of mirroring a get request) and the url extension of the data we want.

def user_data(user_id)     connection.get(’users/#{user_id}’).body end

This, however returns a JSON string. We need to parse it.

def user_data(user_id)     JSON.parse(connection.get(’users/#{user_id}’).body) end

There might be multiple endpoints, so instead of repeating ourselves, its better to create a parse method.

def user_data(user_id)    api_parse(’users/#{user_id}’) end

private

def api_parse(url_extension)     JSON.parse(connection.get(url_extension).body, symbolize_names: true) end

The symbolize_names: true allows us to have symbols as our keys in the hash. We now have our user_data as a nice Ruby hash so we can access the data easily using the keys to get the values we need. 

For our API Curious project, we needed to access and consume a target API for use within our app so we could mimic the site we were getting the API from. 

I chose GitHub as I had already done much of that through Twitter and Twitter seemed to make things very easy.  

GitHub ended up being much more difficult. Their documentation was often difficult to understand and also occasionally flat out wrong! I was informed that making pull requests to GitHub was not a great use of my time, otherwise, I would have tried to fix the discrepancies. 

I also made things harder on myself by deciding to not have a database. This ended up being more difficult than I figured as I found I needed to persist some data in a stateless HTTP.

I ended up resolving the issue by using encrypted cookies to hold a hash with just enough data to rebuild a Ruby object as needed. 

https://github.com/Egogre/LitHub

JSON is one of the most commonly used data serialization formats.

J - Java

S - Script

O - Object

N - Notation

One of the greatest benefits of transferring data using JSON format is that it is readable by both computers and humans. It is very close to like reading a Ruby hash. (In fact to make JSON from Ruby, it usually starts with a hash and then you merely put .to_json after the hash!)

JSON is quickly replacing XML as the standard for API interfacing. XML is much bulkier with lots of unnecessary data. Another nice thing  about JSON is that all JSON is valid JavaScript and... JavaScript is all over the internet! You can manipulate JSON data as you would any JavaScript object. 

When using Google Chrome, there is a JSON Chrome plugin that allows all JSON data viewed to be rendered on the screen in an easy to see format instead of all jumbled together on one line. 

In JSON, keys MUST BE “double quoted” followed by a colon and then the value. 

“key”: value

Values can be Numbers, Strings, Booleans, Arrays, Hashes and Null

Be careful with strings you want to convert to JSON. Do not break any lines without a \n 

JSON is used so often in Rails applications that there is an included JSON library in all standard rails apps (it is automatically required!)

Standard methods are:

.to_json (as shown above)

JSON.parse(JSON_string converted to ruby hash)

JSON.generate(thing_to_convert)

JSON.stringify

Keep in mind that when you parse JSON in a hash, the keys will all be strings and not symbols!

OmniAuthentication (OAuth) allows you to tap into authentication through another site such as GitHub, Twitter, Facebook, and Google. The OAuth gem allows your app to do this through a series of ‘handshakes’ with the other designated app. 

The benefits of this are that you get to draw upon the user security of these larger, well funded applications. 

But, as with all gems, you are relying on an outside source and can only hope they keep their gem updated and keep up with new threats. It can also become complex if you want to add in multiple kinds of OAuth. 

Since you are given access to much of the user’s profile information from the other site, a more conceptual issue arises. Do you update your user’s profile information each time they update their information on the other site? It can get messy.

The way it works is your application redirects a user to the designated authentication site (lets use Twitter as an example) and sends a token request along with your site’s API key and secret. The key and secret tells Twitter that the request is originating from your site and not some fake. 

Twitter then asks the user if it is okay for your application to access the user’s Twitter information. The user must authenticate with Twitter to accept. When Twitter receives the ‘go ahead’, it then sends a payload filled with a bunch of the user’s data to a callback route you set up for your application through the Twitter developer page for your site. Your middleware (Rack) converts this payload information into a hash that Rails can decipher.