Java, Javascript, Python, Linux, Node.js....

A well know aspect of running code on languages which use the JVM is garbage collection. The mechanism of running a background thread to release unused memory and avoid fragmentation.

In Java programs the memory management aspect is abstracted by the JVM. Once we start a jvm process memory is allocated  by the kernel using the virtual memory address that map to User space memory. (Note: I am not sure how much is allocated at start and if this is fixed or increased on demand, will update the same in future edits).

Once memory is allocated to JVM process. Its up to JVM to manage the memory blocks , i.e. allocate and reclaim unused memory.

In this blog I will try to describe the jvm memory model and how garbage collection works along with indicating various jvm parameters that could be used in configuring GC settings. I will also try to mention a few tools that will help in debugging the jvm in production.

Note: This is a WIP blog. I will update this as I learn more.

Java memory model:

The memory model in jvm can be broken into the following components:

MetaSpace

JIT Code cache

Thread Stacks

Heap

MetaSpace:

This region of memory was know as PermGen space in Jdk 7(Permanent Generation space as it would never need any garbage collected).

This is the memory space used to store the loaded class files by jvm class loaders. Since a typical java application tends to link to various libraries, all these would need to be loaded into meta space. This is designed to grow based on need.

But issues would arise if java class loaders end up loading more classes than the physical RAM space. As Linux assigns only virtual memory address to a process, if swap is enabled in kernel options. Some of the memory data in the process with be moved to swap file. This will degrade performance of the java process running on the JVM.

Alternatively jvm-args can be specified to limit the size of meta space. This will restrict memory size and error out when we exceed the limit.

-XX:MetaspaceSize=128m  // specifies 128Mb for metaspace

Note: usually in typical long running server kind of applications the count of class loaded would increase during bootstrap of server and then remain fairly static post that. Exceptional cases will involve loading classes at runtime over network etc which are infact dangerous in security considerations.

JIT CodeCache:

If you recall Java 101 basics. Java takes your sourceCode in *.java and converts it to byte code *.class. This is still not machine specific code. (Machine specific code would vary based on your OS). Byte code is run on the JVM. Hence JVM has a JIT (Just in time) compiler which converts the bytecode to machine specific code, now if jvm determines that a block of code is being frequently accessed it would cache the compiled native code, which will provide benefit of recompiling.

Basic JVM args in setting codeCacheSize: -XX:InitialCodeCacheSize=32m.

ThreadStacks:

Part of JVM memory used store ThreadStacks per thread running in the JVM.

More on this will be updated latter

Heap:

The major part of the jvm memory which is used to store objects.

So when you do:

       String a = “test”;

You are creating a string object “test”.

Which internally is represented by String.class

Where “test” is inside a  char[].

Size of charArray = 2N+ 24; // number of chars  = 4 => 36 bytes;

Total size of the object “test” ⇒ 36 + 16 (object overhead) + 4 (hash) + 4(buffer) = 60 bytes. (The size is an approximate estimate - I could be wrong in the calculation of object overhead and array).

Now 60 bytes need to be allocated by jvm in order to process the line

“test”;

The JVM allocates 60 bytes in the Heap. Also another 8 bytes would be allocated for the pointer “String a” which stores address of object “test”.

Heap size in jvm can be controlled using  -Xms and -Xmx (or: -XX:InitialHeapSize and -XX:MaxHeapSize)

Eg: java -Xms128m -Xmx2g MyApp

Note, this heap is cleaned up by garbage collector enabling us to keep creating new objects and not worry about destroying unused objects and ensuring we are below the MaxHeapSize configure else we would throw OutOfMemoryException.

 The heap is broken down into 2 regions:

Young Gen  

Old Gen (Tenured region)

YoungGen as the name implies is the region where all objects are created and short lived objects exist.

YoungGen is further broken into 2 components.

Eden

Survivor Spaces (S0 and S1)

Eden is where all objects are created first.

Garbage Collection:

There are 2 kinds of GC minor GC and major GC.

Minor GC:

Note some details given below could vary based on type of Garbage collector in use.

Minor GC runs when JVM is not able to get enough memory from Eden.

Minor GC  checks for blocks of memory which are still in use, but running down the Thread stack memory and checking their pointers to memory location, (Note this implies threads need to be paused during this run so that they dont change. Hence minor GC does create a stop the world pause, which would be very small in duration).

Minor GC can be happening on single thread or multithread based on collector applied.

Once it identifies Unused memory spaces in Eden , it runs copy collection. To move all used data to Survivor space so as to clean up the space in Eden.

Survivor space comprises of 2 survivor which are equal in size.

At any given point time one Survivor is ToSpace and other is FromSpace.

So if S0 is ToSpace in the first Run  and S1 is FromSpace and both are empty

In second run S1 is ToSpace which implies both data from Eden which are not collected plus old data in S0 which survived first GC are pushed to S1.

Note: If object is to big and can not be pushed in to S1 or S0 during either collection then it’s pushed in to Tenured region. (This is called premature promotion. One of the issues you can run into if you are not cognizant of your data size and have not allocated sufficient survivor space and eden space)

Post each collection object’s age is increment, hence a long living object will flip-flop from S0 to S1 regions at each minor GC.

If object stays longer than a configured number Tenure, it is copied over to OldGen/ Tenured region, by default this in 15.

We know that go configure heap space to be used we can use

-Xms512m -Xmx512m

Now to configure the portion of Heap to be used by Young Gen we can use

-XX:newRatio=2 -XX:survivorRatio=8  

These options are not quite straightforward and need some explanation.

newRatio = 2 implies that out of total heap allocated we the ratio of OldGen / YounGen == newRatio.

Hence newRatio = 2 implies ⅓ of heap memory will be used for YounGen. In this case close to 171Mb will be used in YounGen and 341Mb for oldGen.

Now survivorRatio=8 implies that each survivor space with ⅛ of the eden. Two survivor space implies ⅛ + ⅛  = ¼

Hence eden would be ¾ of the youngGenSpace.

For the above example that would mean

Eden size would be ¾ * 171Mb = 128Mb aprox

Survivor space S0 and S1 will each be = 21Mb

Note: above options are relative modes of tuning the GC size, Alternatively we can specify specific size for YoungGeneneration which will override the newRatio using

XX:NewSize=171m -XX:MaxNewSize=171m 

XX:MaxTenuringThreshold=15  // To change the tenuringThreshold on which object should move to tenured region which by default is 15.

Major GC:

If tenured region gets filled up then JVM will need to trigger Major GC or FullGC.

System.gc() and Runtime.getRuntime().gc() suggest JVM to initiate GC.

FullGC will remove unused objects in tenured region and will also try to reclaim space for MetaSpace. And for loaded classes which do not have any objects on heap, those classes can be removed.

If metaspace size threshold is provided. FullGC will get trigger to reclaim MetaSpace.

Jvm cmd line tools: (TODO will update this shortly)

jstack

jmap

jps

jcmd

jinfo

One of the common issue in Web service is the problem of network timeout.

As a simple example consider a User registration scenario. User enters all his details and clicks Register button, only to see the web page not responding, He now clicks again and again for a few times and in between one of his click, he is redirected to the Welcome Page.

Looks all good but we may end up having multiple entries of the User in the backend.

There are two approaches to avoid this problem:

Disabling the button till callback of the first click arrives.

To have transactional access to datastore persistence  and use the unique constraint violation to catch the duplicate call.

The first approach is a client centric approach relying that client ensure he calls only once, going to the extent of affecting User experience, Not to mention that backend service system has no defense if there is a loophole in the client code.

The second approach is more robust and forms the base of idempotency handling.

In the world of microservices, this issue is more prevalent, with multiple microservices handling a single user action. As an example consider the example of  buying a product from an e-commerce website.

A single click checkout and but could involve invoking multiple services, such as service to get User address information, service to get serviceability/ availability of the product, service for getting and applying offers, service getting and invoking payment channels and service for successfully storing the order.

The above call flow is a high level description, In reality the call graph could get more complex it could have loops and is not always a DAG, and in these cases the need for idempotency is of a high importance.

Before rushing into implementing all your API’s to be idempotent, It's important to consider when idempotency is needed.

For services which provide serviceability and availability lookup - idempotency is of no use. But services recording order placed, payment request, idempotency is needed to avoid duplicate order/ payment from being processed.

A rule of thumb is if your api is making any Lookup or GET kinda calls, idempotency is not needed. It’s only needed when you're making a PUT/POST call involving create/update of a persistent entity.

Idempotency implementation: (Not as simple as it looks)

An important aspect overlooked while implementing idempotency is the need for transactionality in checking for idempotency.

Let’s consider a simple implementation

if(idempStore.contains(idemKey){

   return idemStore.get(idemKey); // may choose to get or throw IdemAborted exception here

}else{

  -- execute logic

}

This simple implementation has issues of race conditions. The case of 2 concurrent request threads running through the if stateMent around the same time and returning false resulting in multiple execution of logics in the else block.

A simple fix to the above logic is to use a mutex.

sync(idemStore){

if(idemStore.contains(idemKey){

   return idemStore.get(idemKey); // may choose to get or throw IdemAborted exception here

}else{

  -- execute logic

}

}

This would prevent race conditions but increase latency over lock contention.

A few improvements over the above approach would be do all logic outside the synchronized  block and use synchronize only for updating the data in store. Hence contention would be limited to the store operation latency in your application.

if(idemStore.contains(idemKey){

  return idemStore.get(idemKey); // may choose to get or throw IdemAborted exception here

}

-- execute logic

sync(idemStore){

if(idemStore.contains(idemKey){

   return idemStore.get(idemKey); // may choose to get or throw IdemAborted exception here

}else{

               store.udpate(data)

   idemStore.udpate(idemKey);

}

}

Alternative approach to pessimistic locking would be optimistic locking. Compare and Swap strategy, i.e. if this available within your datastore, such as checkAndUpsert in hbase, putIfAbsent in redis. Here the idea is store the idempotency key along with the data and version with CAS update operation and in case of CAS failure where version doesn’t match input version you recheck for idempotency in the data from store and throw idempotencyAbort if the idempotency key exist in the store.

if(store.contains(idemKey){

  return store.get(idemKey); // may choose to get or throw IdemAborted exception here

}

-- execute logic

Boolean success = store.checkAndUpsert(data,idemKey,version);

if(!success) return store.get(idemKey);

I have been working for quite some time in Java but realized that I have never actually used or gone deep into the basic abstractions in Java Threads. With java.util.concurrent package present and providing easy to use abstractions over the fundamental thread model in Java, one doesn't really find the need to use a Thread class and understand its working model. But knowing what goes down underneath sure does give better insights when writing code.

This blog is to recall some of the basic concepts in Java Threads and explains a few higher level abstractions derived from here. Let's start with trying to understand what is a thread.

A Thread is light weight process created with its own copy of local context variables, memory, priority, threadStack size to run independently in the program.

The above might seem gibberish to a few, so let me explain the terms used above in  simpler way:

light weight process -> Similar to unix process the fundamental difference being that each light weight process don't have independent memory , stack segments allocated by the kernel. This is kinda managed by jvm process. Each thread in JVM process is mapped to a light weight process to the kernel so that it can use kernel based scheduling of threads. (Note this is why multithreading behaviour is not standard in different platforms. They depend on how kernel schedules them based on platforms).

priority -> priority is a number from 1 to 10, used to tell thread scheduler importance of a thread and give it more priority when using the processor, more on this when we discuss on Thread scheduling

threadStack -> Each thread during execution of program has a stack to maintain partial computation results. The size of stack can be specified  on thread creation as this is stack is specific to this thread. Note this specific to JVM and not OS.

Thread Types:

In Java there are 2 types of thread. User thread and daemon threads.

JVM is designed to keep running even if one User thread is alive. Daemon are background thread and JVM will quit even if they are running when no User threads are alive.

So when you do public static void main, JVM creates a whole lot of daemon thread for Garbage collection etc.. but one User Main thread to kick start main. And if you were to create more User threads from Main then even after Main thread terminates JVM will run till all User threads terminate.

Thread States:

A thread has 3 Fundamental states:

INIT - Thread has been just created RUNNABLE - Thread is running - (Note.. this doesn’t guarantee this thread is running in the OS processor, it still depends on scheduling more on that latter in this post)

TERMINATED - Thread finished execution. (There is not recovery post termination. A terminated thread is a dead thread)

RUNNABLE thread can get into other sub states:

WAITING   -- when thread has called Thread.wait() method or Thread.join() - waiting for completion of another thread

TIMED_WAITING -- when thread has called Thread.wait(timeInMiliseconds) or Thread.join(timeInMiliseconds)

BLOCKED - When thread is blocked on trying to obtain a lock to a synchronized scope.

Basically all the above substates are more or less similar where thread is not doing anything useful but waiting and may or may not resume its work which by going back to RUNNABLE.

Thread Scheduler:

All the concepts of threads and parallel execution are great, but when it comes to hardware we are limited by number of physical cpu cores we have in our system and parallelism is bound to that cpu core.  (In case of hyper-threading in Intel’s SMT - “Simultaneous Multithreading” design, for each processor physically present the os would register 2 address for cpu cores. But still number of cores is still a small finite number). Fortunately each CPU core runs a high clock rate to execute instruction, The frequency of instruction cycles executed is measured in GHz. So when you hear dual core 2.4GHz, it means you have 2 cores each capable of running 2.4 * 10^9 instruction cycles in a second. Since the cores are so very fast when they can manage many different process by scheduling them and giving each their turn at using the processor for computation. So in case of dual core you can think of 2 lines of queue being sent to the processor. Now comes the concept of which of the threads are more important from the queued list and should be scheduled earlier. This is defined in priority of a thread.

This also means that when a thread is running other threads are waiting. but the time on waiting is small if as processors are very fast.  But if the processors are slow or computation of one thread is very long other threads could wait for a longer time. To avoid this kernels implement time based slicing. where each thread is given a time slice post which next thread is called. if the current thread is not complete in it timeslice it's put on hold and pushed back to the schedule queue and next thread is taken for execution, This prevents thread starvation.

There are two types of Schedulers:

Green Scheduler - JVM based - no Time slicing

Native Scheduler - Kernel Based (Time Slicing)

By default the native scheduler is used to take advantage of kernel time slicing , but this makes multithreaded application platform dependent in terms of scheduling behaviour.

Now when does a scheduler switch from one thread to another. There are 4 cases:

the running thread finishes in its timeslice

the running thread is not finished but time slice is used up.

the running thread does a blocking IO and is waiting for network server to come back with IO response.

A higher priority thread wakes up from IO response and the current thread is preempted (software interrupted) to go back to queue and the higher priority thread takes the processor.

Note: This is why is NonBlocking Java NIO based servers, IO threads are set at higher priority than Task worker threads. Cause we want IO to keep running and accepting connection and not be blocked because worker are doing high intensive calculation and taking more of processor time.

Thread Methods:

sleep, isAlive, join, setPriority, yeild, synchronize, volatile, ThreadLocal, wait, notify, notifyAll, interrupt(), interrupted(), isInterupted()

The above methods help working with Threads:

Thread.sleep(timeInMilli); // used to put a thread to sleep

isAlive() ; // instance Method that returns boolean indicating if Thread is RUNNABLE

A combination of above 2 is used when you want master to create child thread and then wait for for child to complete before it terminates;

Thread childThread = new ChildThread();

childThread.start();

while(childThread.isAlive()){

Thread.sleep(200);

}

join(); - when called on a the child thread waits for that child thread to finish before running through.

So essentially is simplifies the above code:

Thread childThread = new ChildThread();

childThread.start();

childThread.join(); //will wait till childThread is completed

//other variants include join(timeInMilli); wait till completion or time

Another use case of join is indefinite wait

Thread.currentThread().join(); // calling join on itself is an indefinte wait

This is better alternative to

while(true){

  Thread.sleep(100);

}

setPriority(int arg);  arg takes values 1 to 10 MIN_PRIORITY, MAX_PRIORITY and NORM_PRIORITY.

It needs to be specified before starting the child Thread. This helps to influence scheduler as mentioned above.

yeild() is a simpler way to influence scheduler but has not guarantees.

Calling yeild(); will interrupt the calling thread and push it back into queue and pick the next thread in queue based on priority , it could turn out the same thread is picked. hence no guarantees.

synchronize

This keyword is used to serialize access to a scope in code defined by this block. But care should be taken to understand what is the monitor on which the scope is locked.

synchronize establishes a monitor on the object on which it is called. Think of monitor as a field in the instance/class which only one thread can hold a lock .. and other threads are queued.

synchronize(obj){

//in this scope only one thread holding monitor to obj will run

}

synchronize int method(){

//in this instance method monitor will be established on instance object for which this method is called

}

synchronize static int method(){ // in this monitor is on class object of the class of this method. }

Points to keep in mind when using synchronize:

Synchronization is expensive.. time is taken to lock and release

Avoid having nested synchronized methods. one method calling another and both synchronized. (These lead to deadlocks and race conditions)

synchronize construct is good at doing its job but is not flexible enough.  Some of its shortcomings are:

Threads waiting to acquire lock are indefinitely waiting and cannot be interrupted.

Also in case of nested locking lock they mostly run into dead locks and synchornize is scope based and lock is not released post scope and there are no failures.. if second nested lock is acquired by some other thread, we wait indefinitely and not fail fast.

Alternatively Java-1.5 introduced Lock interface and its implementations such as RentrantLock, ReadWriteLock etc.. which provide greater flexibility. More on them in a separate blog post. In this I will continue to focus on the basics.

volatile:

Each process fork in OS has its own copy of its memory, but in JVM a thread which is a lightweight process has isolated memory inside the JVM heap all local variables created inside the new Thread are local to itself. Global variables called by more than one thread are shared. But while executing operations each thread would store the intermediate computation result of these from the processor into the cache register. This leads to corruption of data. To avoid this volatile keyword is used. volatile keyword ensures that when one thread changes the value of this variable its flushed back to RAM - JVM heap. so that other registers know about it. The way this is implemented is by adding additional instruction to processor. Each processor gets a Flush instruction to execute post manipulation to the variable, which enforces the flush. This design is called Memory barriers - Load and Store barrier instructions are used for this purpose.

The visibility problem does not occur when threads use synchronization to access shared field variables. When a thread acquires a lock, the thread's working-memory copies of shared field variables reload from their main-memory counterparts. Similarly, when a thread releases a lock, the working-memory copies flush back to the main-memory shared field variables.

Note: volatile variable does not do synchronization, Consider the below code

volatile String test;

t1 = new Thread(new Runnable(){

  public void run(){

      test =”Data”;

      if(test != null && !test.isEmpty()){

//do something

      }

}

t2 = new Thread(new Runnable(){

  public void run(){

      test = null;

 }

t1.start();

t2.start();

// Here t1.start is scheduled first

// it executes test=”Data” which is flushed and exposed to thread t2 aslo

// now t1 executes test != null  which returns true.

// not t2 gets its timeslice of processor

// t2 executes test=null which is flushed and exposed to thread th1

// not t1 get its timeslot

// t1 executes test.isEmpty() -- which throws NullPointer.

Hence from the above example we can see that synchronization and volatile are very different things and should be used appropriately based on what we are trying to do. volatile works well for single shared field with read write operations whereas synchronization is needed when there are either more than one fields or same field with multiple steps of operation as in our example code above where nullCheck  is one operation and isEmpty was the other.

ThreadLocal & InhteritableThreadLocal:

ThreadLocal is a very useful abstraction that enables storing values per thread in the application. Primary use cases is in web applications to store request context per thread.

This is also used in J2EE frameworks to simulate session object using cookie based session that is passed on to request context.

Basic usage of ThreadLocal

static ThreadLocal<SomeClass> context = new ThreadLocal<SomeClass>(){

      @override

   protected synchronized Object initialValue(){

//initial value of this threadLocal

  }

}

Note: initialValue is not mandatory to be overridden but helps in creating a ThreadLocal with default initialValue.

Other methods include:

Object get()

void set(Object value)

ThreadLocal reference are always static as multiple threads accessing a static shared object is the only place ThreadLocal is useful as it avoid synchronization as each Thread has its own store of value. Infact ThreadLocal is not a datastructure that stores, It stores value in the respective Thread.class that invokes it. It provides a pattern of accessing data from Thread.

More on internals of ThreadLocal will be written in a separate Blog.

When building applications where a single request to server spawns multiple threads, it’s important to understand that the request context stored in the ThreadLocal is lost. As new threads created do not have access to it. To avoid this a common practice is clone the object and initialize it in the new Thread starting lines to update ThreadLocal.

A better and elegant alternative is to use InhertiableThreadLocal. This inherits the Threalocal Context values of parent threads to its child threads. Note the work inherits hence child threads can override this value just like in inheritance.

wait() and notify() are methods of Object hence any thread running can call this on any object in java.

calling wait() instance method cause the thread calling it to go to WAITED state.

calling notify() instance method on this same object cause the last thread that called WAIT to wake up and start running.

A use case of this is in Publisher Subscriber model using shared object where you want to synchronize publisher with subscriber accessing a shared stated. I am not going to write that code down as it's very easily found on the internet.

There are other variants of this wait(time) and notifyAll() -- wake up all threads that had called wait.

Note: wait() and notify() should always be called inside synchronized scope else this will lead to ilegalStateMonitorException. A thread must hold the monitor lock on the object on which it intends to invoke wait() or notify().

This is important cause only one thread can call wait() and notify() at a time to have predictable behaviour. If multiple threads are waiting and notifyAll is called by other thread, All threads wake up but are queued up and run through the synchronized scope one by one.

interrupt();

This is an instance method of Thread that can be called on by another thread. Its purpose is interrupt() a waiting thread and throw ThreadInteruptedException if it was waiting.

When interrupt() is invoked refrenceObject of a thread running, two things can happen:

If thread is running without sleeping or waiting or join  it will update the running thread boolean state variable interrupted to true, hence isInterrupted() will return true

If thread was in sleep, wait, or join , it will throw ThreadInterruptedException, (Note: here interrupted state variable remains false

One way to check if a thread was interrupted is using isInterrupted(); another way to check is

interrrupted(); but this clears the state and makes it false

Thread Hierarchy and Grouping:

Grouping of threads to provide certain common characteristics among them such as setting max priority that the threadGroup can have etc..

By default Threads in Java have thread Groups hierarchy. The top most Thread Group is system. system has many daemon threads and subThread group main. main has thread main executing public static void main. Now main thread could create sub thread groups or sub threads.

ThreadGroup a1 = new ThreadGroup(“a”); //creates a threadGroup under main threadGroup as main thread is executing this

ThreadGroup b = new ThreadGroup(a1,”b”); // creates b threadGroup under a threadGroup

Thread th = new Thread(a,”thread1”); // thread 1 created under “a” threadGroup

a1.activeGroupCount(); // gets list of threadGroups under a which woudl be 1

Thread.currentThread().getThreadGroup().activeCount(); // will return 2

Thread.currentThread().getThreadGroup().list(); // print to screen.

print notation:

for threadGroup

java.lang.ThreadGroup[name=<threadGroupName>, maxpri=<maxPrioritySetForThisGroup>]

   Thread[<threadName>,<priority>,<threadGroupName>] //thread under this group

   //similarly subGroup here

//peer groups here

Note: A thread created inside a threadGroup with maxPriority 5 can never be more than 5. Even if we create and set its prirority > 5 its automaticaly reduced to threadGroups max.

To set ThreadGroup’s max priority use tg.setMaxPriority(val) method.

Note: if threads where created in this group prior to this method. They have no effect.

ThreadGroup enable interrupting group of threads which are waiting .. using

tg.interrupt();

ThreadGroups enable simple enumeration of threads:

Create an array of size specified by tg.getActiveCount() // number of threads or

tg.getActiveGroupCount(); // number of threadGroups

update array using enumrate methods

int enumerate(Thread[] thdArray); Get all threads including its subgroup threads.

int enumerate(Thread][ thdArray, boolean recurse); // use recurse false to not go to subgroups

similar methods for ThreadGroup[].

When deploying a new version or a complete rewrite of an old system a lot of things can screw up.  Here I am documenting my learnings on carrying out deployment of user facing systems in ecommerce space where such systems are expected to be Highly Available with zero down time. In short I would say,  Make sure you have a proxy layer.

As you make major changes to your system, it's bound to impact on your contracts with other systems. Consider a case where you may choose to scope down a single system to smaller independent microservices, in such cases your api endpoints also change significantly. Building a proxy which acts as router and also an adaptor to translate request to your contract specification is very useful, this way you can deploy independently and give a deadline to other systems to move to new contract post which you shutdown the proxy.

Another major advantage of using a proxy layer is being able to test and onboard the new system easily. Consider a scenario of rewriting a mission critical system in user's path. In situations where the system is very complex with many use cases, And there is neither sufficient stage setup up to test nor the time to spend on stage testing. Added to this is the fact that the new system is completely new such that  its data model is redesigned, its data store has been changed, it's been broken into smaller micro services. In such cases having the outside world talking to the old system over proxy helps running the new system in passive mode and being monitored.

To run in passive mode the trick would be to use proxy layer to fork incoming request into two request - one hitting the main system and the other the new system. The proxy also ensures that output of the first is sent back to client and the output of second is logged along with the main system output. Now the output of both systems can be checked for diff by a cron for any anomalies.

How does this Help. Well imagine running passive mode, it will start to spit out all errors in the system, which you can keep fixing and deploying. Run it for long enough and all your new data will be auto migrated as passive mode had been creating entries in the new systems. The outside world is oblivious to the fact you had 2 system running and when you configured proxy to cut over and pointing only to the second system.

Having said this, there is more to this then using a proxy to fork incoming request. There is also outbound request where things get challenging. As outbound call by primary system and secondary can occur at different points of time. This makes the job of anomaly detection in outbound calls difficult.

Here the primary problem is how to understand which request payloads to compare when they are serialized and one way to solve it is to know what field in the payload to group on, Say both request from old and new system would have primary key like an entity id. But this does not work always and it’s not the right way as now your anomaly detector system is specific to you business logic and coupled to it. It didn't for us, as we were even changing the format of ids from int to string. The better way was to use meta data such as request Headers in case of Http outbound request, This is similar to concept of request tracing in microservice architecture.

If you have heard of zipkin @ twitter you will know what I mean, else don't worry I will try to provide a gist of what it is in my context. It's the idea to monitor and measure metrics across systems in a microservice architecture. Consider one request to web page at facebook hitting 50 services in the backend. Now If you want to analyze a sample of such request and see the duration it took from each sub system, as tree from the user clicking on the homepage to understand bottlenecks.

On a high level this is achieved by using headers (meta data) to pass in a trace Id identifying the request across the system. This can only work when all systems agree to pass this meta data in their outbound calls. In big companies this get standardized by implementing a generic reusable outbound proxy such as finagle in twitter, to some extent service proxy in flipkart. Hence leveraging a similar traceId generated by our proxy which forks incoming calls to the old and new system we can catch the successive outbound calls made by both systems and group then on the meta trace id.

Reference notes to understand Clojure by translating it to Java from a existing github clojure project (https://github.com/gerritjvv/tcp-driver). Note this is a WIP could have a mistakes especially in types and generics interpretation, but will give you an idea of how go about coding in clojure project.

Note: If you are new to clojure read the earlier post.

##########################################################

Commenting

;; is similar to // comment in java

##########################################################

Namespacing

ns is used to namespace similar to package in java

(ns

 ^{:doc "TCP connection pools

         see: create-tcp-pool"}

 tcp-driver.io.pool

 (:require

   [tcp-driver.io.conn :as tcp-conn])

 (:import

   (org.apache.commons.pool2 KeyedObjectPool BaseKeyedPooledObjectFactory)

   (java.net SocketAddress)

   (org.apache.commons.pool2.impl GenericKeyedObjectPool GenericKeyedObjectPoolConfig)))

Above is similar to:

/**

* TCP connection pools

* see @link create-tcp-pool

*/

package tcp-driver.io.pool;

// This line is specific to clojure import

// Imporing another clojure file and namespacing it

 (:require

   [tcp-driver.io.conn :as tcp-conn])

 // here conn.clj is another clojure file imported as tcp-conn object.. which as public methods that can be accessed

import org.apache.commons.pool2.KeyedObjectPool;

import org.apache.commons.pool2.BaseKeyedpooledObjectFactory;

import java.net.SocketAddress;

import org.apache.commons.pools2.impl.GenericKeyedObjectPool;

import org.apache.commons.pools2.impl.GenericKeyedObjectPoolConfig;

##########################################################

Interface

protocol is similar to interface in java

(defprotocol IPool

 (-borrow [this key timeout-ms])

 (-return [this key obj])

 (-invalidate [this key obj])

 (-close [this])

 (-num-idle [this] [this key])

 (-num-active [this] [this key]))

above is similar to

public interface IPool {

   void borrow(key, timeout-ms);

   void return(key,obj);

   void invalidate(key,obj);

   close();

   IPool num-idle();

   IPool num-active();

}

##########################################################

Class

eg of simple class

(defrecord HostAddress [^String host ^int port])

this in java is:

class HostAddress {

   String host;

   Integer port;

   public HostAddress(String host, int port){

       this.host=host;

       this.port=port;

   }

}

eg of class implementing interface

(defrecord KeyedTCPConnFactory [^GenericKeyedObjectPool pool]

 IPool

 (-borrow [_ key timeout-ms] (.borrowObject pool key (long timeout-ms)))

 (-return [_ key obj] (.returnObject pool key obj))

 (-invalidate [_ key obj] (.invalidateObject pool key obj))

 (-close [_] (.close pool))

 (-num-active [_] (.getNumActive pool))

 (-num-active [_ key] (.getNumActive pool key))

 (-num-idle [_] (.getNumIdle pool))

 (-num-idle [_ key] (.getNumIdle pool key)))

this in java is:

//Note key does not need to be string.. making that assumption as of now

class KeyedTCPConnFactory implements IPool {

   private GenericKeyedObjectPool pool;

   public KeyedTCPConnFactor(GenericKeyedObjectPool pool){

       this.pool = pool;

   }

   public Tuple borrow(String key, int timeout-ms) {

       return new Tuple(pool.borrowObject(key),timeout-ms);

   }

   public void return (String key, Object obj){

       pool.returnObject(key,obj);

   }

   public void invalidate(String key, Object obj){

       pool.invalidateObject(key,obj);

   }

   public void close(){

       pool.close();

   }

   public int num-active(){

       return pool.getNumActive();

   }

   public int num-active(String key){

       return pool.getNumActive(key);

   }

   public int num-idle(){

       return pool.getIdle();

   }

   public int num-idle(String key){

       return pool.Idle(key);

   }

}

##########################################################

Basic function declared in clj

(defn borrow

 "

 Params:

  pool an instance of IPool

  key an instance of tcp-driver.io.conn.HostAddress

  timeout-ms long timeout in milliseconds

 Exceptions: NoSuchElementException, Exception"

 [pool key timeout-ms]

 (-borrow pool key timeout-ms))

here borrow - function name

"  text here is java doc"

[pool key timeout-ms] are args

(-borrow pool key timeout-ms) // body of function i.e. execution

this execution says call -borrow (note - prefix indicates private or member method of pool (the first arg after this)) on args key and timeout-ms

i.e. pool.-borrow(key,timeout-ms); // which is defined

this function borrow is global declaration and can be used by other clj files importing this file as mentioned in

 (:require

   [tcp-driver.io.pool :as tcp-pool])

(tcp-pool/boorow pool key timeout-ms)

Note to declare functions in a type safe way to specify output types

(defn

 ^InputStream

 input-stream [conn]

 (-input-stream conn))

here first arg with ^ symbol prefix specifies output type

followed by method name

then optional "<doc here>" doc which is absent in above example

followed by args

then a body which says call conn.-input-stream();

##########################################################

To Summarize *.clj file start with name-spacing and import followed by interface if any, class if any then global variables if any and public methods

eg. conn.clj

;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;names space declaration and imports

(ns

 ^{:doc "TCP Connection abstractions and implementations

         see host-address and tcp-conn-factory"}

 tcp-driver.io.conn

 (:import

   (java.net InetAddress Socket SocketAddress InetSocketAddress)

   (org.apache.commons.pool2 BaseKeyedPooledObjectFactory PooledObject KeyedPooledObjectFactory)

   (org.apache.commons.pool2.impl  DefaultPooledObject)

   (java.io InputStream OutputStream)))

;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;Class HostAddress and ITCPConn interface

(defrecord HostAddress [^String host ^int port])

(defprotocol ITCPConn

 (-input-stream [this])

 (-output-stream [this])

 (-close [this])

 (-valid? [this]))

;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;Class SocketConn implementation of ITCPConn

(defrecord SocketConn [^Socket socket]

 ITCPConn

 (-input-stream [_] (.getInputStream socket))

 (-output-stream [_] (.getOutputStream socket))

 (-close [_] (.close socket))

 (-valid? [_] (and

                (.isConnected socket)

                (not (.isClosed socket)))))

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

;;;;;;;;;;;;;;;; Public API exposed from this clj file

(defn wrap-tcp-conn

 "Wrap the Socket in a ITCPConn"

 [^Socket socket]

 (->SocketConn socket))

;;; here body (->SocketConn socket) is as good as return new SocketConn(socket); instantiation and return

(defn create-tcp-conn [{:keys [host port]}]

 {:pre [(string? host) (number? port)]} ;;TODO figure out how this works, appears to be validation checks

 (->SocketConn      ;; return new SocketConn

   (doto (Socket.)  ;; instantiate new Socket and

     (.connect (InetSocketAddress. (str host) (int port))) ;; call connect on new InetSocketAddress with args and

This post contains adhoc notes I took while listening to talk introducing clojure. It expects you to know programming concepts well preferably Java.

Fundamentals

Dynamic - dynamically typed

a new Lisp, not Common Lisp or scheme

more readable than Lisp

flexbility (dynamic type system more productive for developer),

Interactivity (REPEL),

Concision (Java not concise),

Exploration,

Focus on your problem - removes all un necessary syntax

static languages like java are inherently slower and keeps devloper productivity less

java is not concise

static languages are usually more concrete which is good is some places

Other dynamic languages for JVM:

Lot of dynamic languages writeen for others like CPython of Cruby have some libraries writtern in native C for performance

and having them in java requires them to ported to java

- Ports to JVM : no written for JVM.. just ported to JVM i.e. need to write clibrary to JVM to mimic Cpython or Cruby

- JRuby

- Jython

- Native JVM: witten for JVM

- Groovy - dynamic nature to java

- Clojure - changes style of programming

Functional

- emphasis on immutability

Hosted on the JVM

Supporting Concurrency

- langauge support to avoid complexities of locking model

OpenSource

Syntax and evaluation model

Sequences

JavaIntegration

Concurrency

Why Clojure:

Clojure is useful for

- scheduling

- broadcast automation

- projection

- listening and analysis system

mathematical elegance of lambda calculus

Expressive elegan

Good performance

 - useful for the same tasks java is

 - Wrapper free Java access

Powerful extensibility

Functional programming and concurrency

Clojure is a Lisp

- Dynamic

  -REPL

   eg (+ 1 2 3) => 6

  -Define functions on the fly

  -Load and compile code at runtime - (No distinction -- compile  happens all the time)

  -Introspection

  -Interactive Env

- Code as data

- Reader - interface of text to evaluator

- Small core - very small syntax

- Sequences -abstract data to different data sturctures

- Syntactic abstraction- ability to suck out repetition

Atomic Data Types:  -- All numbers are Objects no primitives

 Arbitary precision Integers: -123456789 // can get as long memorys supports

 Doubles 1.34 -- java Doubles.. (Not primitive.. everything is an object)

 BigDecimals 1.23M

 Ratios - 22/7 // hence you dont lose info for irrational number (Frink a language for JVM for precise computation without loosing info)

 Strings - "fred"    // imutable same as java

 Characters - \a \b \c

 Symbols - fred ethel, // identifiers - first class object like strings

 Keywords - :fred :ethel // very simlar to symbols which evaluates to itsefly.. symbols can be mapped to other object values

  - very useful in maps as key as they are fast for comparision..

 Boolean - true false

 Null - nil // nil is false is the same value as java Null .. so you can use if(nil) which will eval the java branch

 Regex patterns #"a*b" - moves to compile pattern that can be used in macro

 functions on sequences will return nil on empty list.. hence if condition would work..

 what should be difference between nill, Null and empty list. (None in Lisp, some in scheme and mix of both in Clojure)

Data Structures: (Aggregate Datastructures)

All are immutable and persistant data structures - (recall temporal datastructures)

commas inside list are treated as white space in clojures. ignored by clojure

Lists - singly linked - grow at front - usage like stack

(1 2 3 4 5), (fred ethel lucy), (list 1 3 5) -3 different list, final list has one symbol and other integers

Vectors - indexed access, grow at end

[1 2 3 4 5], [fred ethel] - same as array.. append new elm at end

Maps - key/val associations

{:a 1, :b 2, :c 4}, {1 "ethel" 2 "fred"} // same as hash map.. there are also sorted maps

first is keyworks and integer , second is integer and string

Sets #{fred ethel lucy} // hash sets.. useful for contains..  there are also sorted sets

Everything Nests

test for equality == //similar to python.. only compares values

- closure demphaises identity.. only compare values unlike java

Cloure implements egown -- immutability

anything can be key or value.. a key can be a vector or map also

having complex structures as keys could be time consuming in calculating hashcode... but then the these hash are usually cached..

having tupples as keys is usually very useful

There is no locking needed for these datastructures in concurrent programming..

the add times are a bit higher than standard. but this has better lookup time.. beacuse of better cache locality

There is destructuring.. not pattern matching.. i.e. breaking down and matching lower levels

Syntax

Data structures are the code

 - Homoiconicity

Structure of clojure is a series of datastructures

Representation of datastructures define the program

No more text-based syntax

Syntax is interpretation of data structures

Traditional Evaluation:

Code Text -> Compiler (Syntax tree, parser, lexer, synatx rules) -> bytecode -> Executable class/jar - Run Java

- JVM - effect

Clojure Evaluation:

 Code Text -> Reader (take chars and create datastructure for jvm)-> evaluator/compiler (never sees text only datastructures)-> bytecode -> jvm (since its interactive there is not javac.. this direclty executes) -> effect

 what comes out of reader is datstructrues..

 evaluator compiler never sees text.. it only sees datastructures..

 clojure comes with builtin compiler and its interactive

 datastructures in clojure is written in java.. some operators are written in java.. and most of clojure is written in clojure

 seperation of reader allows you to change file text to user input to reader.

 Its also possible to write some other program that generates datastructure that is sent to eval/compiler

 program -> eval/compiler - jvm - effect

 REPEL - reader - eval/compiler -> bytecode - jvm - effect

 or have program writing pogram by communicating with compiler/evaluator

 program - eval - bytecode - jvm - effect

   java you have no ability to shape the language

   in lisp you have a way to modify the jvm behaviour by introducing a macro..

   the idea is you have a program interacting with eval/compiler

   this program says if eval/compiler sees a datastructure which this program/macro wants to overrride.

   it is sent to the program/macro which eval ans sents it back to eval/compiler which then computes byte code and pushes to jvm for effect

 program/macro - eval/compiler - program/macro

this way the language is powerfull to enable user to define behaviour of the language through macro constructs and providing syntactic extensibility

There are namespace over here to have different functions with same name

 clojure is writeen in java

Expression

In java there is statements, declaration and expression

in clojure everythin is an expression

- All data literals represent themselves:

 exceptions are:

  symbols = looks for binding values first localy then globally

   let - local symbol creation

   def - global symbol creation

  list = an operation form

Operation forms:

(op ....)

- op can be either

- one of very few special ops

- macro

- expression which yields a function called using calling mechanism of clojure

Lisp1 supports macro well - Clojure uses this

note symobl can be used as place holders to function and used here

when symobl say add is used. its first checks if there is a macro handler registered for add .. if not it checks if add is a symbol

and what does add refer to .. and evaluates the function expression it refers to

Special Opst:

Core of computation can be built of just 7 primitives

can have non-normal evaluation fo args:

 (def name value-expr) - def does not evaluate name.. it evals value-expr to name

  establishes a global variable

 (let name value)

   establishes a local variable

 note: name can only be a symbol

 def defined variables are global and can be redeclared.

 def is not set

 symbols are not inherntly in namespace. they are just variable.

 there are var have places

 (if test-expr then-expr else-expr) - function eval all args.. hence if is special and evals just one of its args. based on first args

  - condional evalustes only one them

 fn let loop recur do new . throw try set! quote var // these are other special functions

 above all are part of defmacro thats bootstraped in clojure to define these special functions

 in clojure functions are macros

 above are all the special functions in clojure

Macros:

Supplied with clojure, and defined by user.

 - empowers user to extend language

when you write a macro.. and once clojure evals the first opeartor to be macro you its send the call to macro program with data. and expects data structures in return.

macro could do db lookup or rule engine logic etc.. in any logic and sent back in sockets if needed

and this keep going.. i.e. macro could give macros..

no this during compilation before byte code.

note: or is not primitive in clojure.. its built on if

  argumens are passed as data to the macro function which returns a new data structure as replacement of the macro call

  (or x y)

   becomes (let [or__158 x] (if or__158 or__158 y))

    let assigns or_158 symbol to value of x post execution - this ensures x is evaluates only once

    then if implementation is true returns the value..

    no all values can be placed in if condition and values which are nil/false are only false.

    let takes n args.. i.e. a vector of name val etc.. so here its only one in the vector

    note : all expression are bound

    BigBoolean in java is not stable.. so it BigBoolean not false could be false in conversion to clojure

So macro takes over where special operators leave of... hence everything is built over the special opeartions

So errors when you look at stack trace.. is difficult when something break in macro

macro enables redundancy of repeat code like closing in java

IDE may enable auto generation of repeat code. .. but fixing error requires modification to all areas of the code..

So with macro the boon is single point to build this abstraction and reuse.. but also single point where it would fail across the system.

Functions

- First class values (Methods in java are not.. you cant method to variable.. in clojure you can)

(def five 5)

(def sqr (fn [x] (* x x))) // fn creates a function object that is evaluated when sqr symbol is called

(sqr five)

25

now if you give two args you get invalid arg exception

Maps are function of their keys

(def m {:fred :ethel :ricky :lucy}) // define map of keywords in keys and values

(m :fred) // get call of key is like function execution with key as arg

:ethel

Syntax summary

int i= 5;   (def i 5)

if (x == 0) return y else return z   (if (zero? x) y z) // zero? is function name

x * y * z ;     (* x y z)

foo(x,y,z);    (foo x y z)

foo.bar(x);    (. foo bar x) // for interoperability  the dot tells clojure we are doing java stuff

there is tremendous uniformity through out the language

Sequences:

Abstraction of tradional Lisp lists

(seq coll) // getting sequnce from collection return nil if collection is empty

(first seq) //return first element

(rest seq) // return rest of list or nil if empty

this abstraction of sequnces to give first and rest is bound to list in lisp but apples to vectors and maps as well in clojure

this lets manipulating datastructrues without loops

Note this is similart to scala list structure

Sequence Library:

// below constructs are built on sequences.. first and rest

(drop 2 [1 2 3 4 5]) -> (3 4 5) // could be vector or string or anything.. as seq abstraction over that works with first and rest

(take 9 (cycle [1 2 3 4 5])) -> (1 2 3 4 5 1 2 3 4) // you can build infinet sets with sequences

here cycle returns an infinite list and take 9 takes only 9

so here call to rest will generate rest + first one added.. this makes infinite sequence

you can use this for roundrobin distribution

(interleave [:a :b :c :d :e] [1 2 3 4]) -> (:a 1 :b 2 :c 3 :d 4)

// one these could be infinite..

(partion 3 [1 2 3 4 5 6 7 8 9]) -> ((1 2 3) (4 5 6) (7 8 9))

(map vector [:a :b :c :d :e] [1 2 3 4 5]) -> ([:a 1] [:b 2] [:c 3] [:d 4])

// map calls vector function of :a 1 then :b 2 and so on..

(apply str (interpose \, "asdf")) -> "a,s,d,f"

// apply applies str function with args which are sequences

// interpose says take , char \, and put it in between every sequence of data 'asdf'

// so apply makes sequnce output  to args of str function

(reduce + (range 100)) -> 4950

// reduce applies + to successive paris of its arguments

Java Interop

(. Math PI) // . says next would be java

// first checks if next args is class or instance

// if class its a static field or static method

// if not class its instance

3.14...

(.. System getProperties (get "java.version")) // says use . between every fields (this is a macro which recurse to give a nested .)

// Here System getProperties gives Properties

(get "java.version") is the get method call on properties with args "java.ersion"

// .. is an oridnary macro

// defmacro - used to define macro .. it takes forms and returns forms

"1.5.0_13"

(new java.util.Date)

Thu Jun 05 ..

//doto is a macro

above create Jframe as frist args to all other args

(doto (JFrame.) (add (JLabel. "Hello World")) pack show)

// expands to

(let* [G__1837 (JFrame.)] // make new jframe

 (do (. G__1837 (add (JLabel. "hello world"))) // add new label to this Jframe

     (. G__1837 pack) // call pack

     (. G__1837 show)) .. call show

 G__1837)

Above in java would be new Jrame().add(new JLabel("Hello World").pack().show();

Java Integration

Cloure string are java strings

numbers are Numbers

collections implement Collection // collections of clojure are interable and use readonly part of java Collection

fns implement Callable and Runnable

Core abstraction like seq are Java interface.. Hence custom datastructure implements ISeq will have all the benifits of using sequence library

Clojure seq library works on java iterables Strings and arrays

All Clojure datastructures can be extended in java interfaces

New pirmitive artihmetic supports equals javas speed

Been long time since I posted. I picked up a lot of new learnings on the job and never had time to write it down. Hence starting up the coolest thing I learnt.  This post is related Graph Algorithms, if you have never heard of Graph Algorithms, this post is not for you. The post assumes that reader is aware of Graphs, Nodes, Edge, etc..

Vertex Coloring, The Problem:

Given a Graph with V nodes and E edges, What is the minimum number of colors we need to color the graph such that no two adjacent nodes (nodes with common edge) have the same color.

In Algo and DataStructures the Vertex coloring problem is an NP Complete problem. But in most practical cases the problem can be solved linear time. In fact incase of distributed computing it could be solved in Log*n.

 There are good post out there explaining this Algorithm.  I am not going to get into explanation of this Algorithm but rather focus on its applications at work.

Event Scheduling:

The process of scheduling different events in a given time range with as effective use of time as possible.

A text book example of this is exam scheduling . Consider

V (Nodes)  - Subjects

E (Edges) - ( connect subjects with common students i.e. student taking both subject)

Color - Time slot for the exam in a bounded time range (say 1 day)

The min number of colors to color the Graph here is the minimum number of time slots needed to conducted the exam of all subjects.

If you can see the Gist of the idea here.. To look at this from Parallel processing problem:

V (Nodes) - Independent Tasks (Process, or LWP/ Threads in an Java process)

E (Edges) - Common resources consumed by the 2 process. (not thread safe)

Color - Execution Time slot for each process 

Now one cool application I found that can be used at work is to look at this when designing an architecture  to scale systems built over micro service architecture with many services.

V (Nodes) - Microservice process

E(Edges) - Asynchronous connection or no connection between Microservice systems

Color - Scale tag for a mircoservice - (WebScale or Not)

Note: 

There are 2 modes of communication between the Microservice systems

 1) Sync - REST request etc..

 2) Async - Queue etc..

The edge is considered as Async or no connection as systems connected with sync call need to have similar scale requirements.

Output graph will help distinguish between systems that need to WebScale , Highly Available (HA) and other systems which do not need to be.

This also gives better clarity on systems which are the bottleneck in the microservice architecture i.e. systems in same color with lesser scale.

This blog post just summarizes my understanding of the framework. Refer documentation to see snippets to understand the notes below better.

Main features:

1) EventDriven Non-Blocking:

Locust helps achieve concurrency of higher rates than that can be achieved when using multiple threads or light weight process as in java. As each LWP uses some ram and the concurrency is limited by ram and cpu. The CPU’s IO wait time increases and chokes the load average of the system.

In case of locust its built on gevent which in turn is built using greenlet and is tightly couple libevnt kernel lib (/dev/epoll) that propagates events. This event driven approach enables multiple task to execute in a single thread. Similar to javascript single thread event based loop execution.

Hence a greater concurrency or request can be achieved on smaller configuration of ram and cpu.

gevent over greenlet wraps callback structure of libevent to simpler synchronized constructs that can be used to make programming simple and avoid callback hell.

Note: More info of getevn and greenlet will be in separate blog post

2) RealTime metrics:

Locust also enables realtime view of request and response outcome, throughput etc.. unlike other test framework where post test is when we find reports of the test.

Installation:

Note: Ensure you have libeven installed

pip install locustio

or 

easy_install locustio

Note: using virtualenv is better to work with different lib and version of python

Distributed load testing in locust is achieved by pyzmq (ZeroMQ)

The intercommunication across test running in different boxes is through pyzmq

Distributed mode involves master running just the UI and orchestrating the slaves rather than actually running the test. master takes care of dividing the load and number of users across slaves equally and collects statics from the slaves

Master runs a server on its hosts on port 5997(default can be changed) and the slaves when started are specified the host and port of master to connect to

starting master and slave is as simple as

locust -f <path to locust file> --master

locust -f <path to locust file> --slave master-host=<master host ip> 

nproc limits in linux are set in /etc/limits.conf

nproc - returns number of available processors

nproc is defined at OS level to limit the number of processes per user.

the way to set nproc limit:

ulimit -u 1024 # set 1024 process per user

Note: ulimit -a list all system variables.. 

ulimit -n => number of opern files allowed

ulimit -u => max process per user 

there other variables like stack size cpu time etc.. 

In linux when doing mulithreading each thread is impletmented as light-weight process (LWP) and to see the Light weight process using ps we need -L option

ps h -Led -o user | sort | uniq -c | sort -n  # gets light wegiht process count per user

Note: h removes the header line

The count per user should be less than nproc limit.

another way to show number of threads per process

ps -o nlwp,pid,lwp,args -u user | sort -n

The above command list out grouping of thread used by each process by the same user as user can run more than one main process

There are many ways to do this, But the cleanest approach is to install the dmg of all the versions your need and modify .profile with the following aliases:

export JAVA_8_HOME=$(/usr/libexec/java_home -v1.8)

export JAVA_7_HOME=$(/usr/libexec/java_home -v1.7)

export JAVA_6_HOME=$(/usr/libexec/java_home -v1.6) 

alias java6='export JAVA_HOME=$JAVA_6_HOME'

alias java7='export JAVA_HOME=$JAVA_7_HOME'

alias java8='export JAVA_HOME=$JAVA_8_HOME'

Note here /usr/libexec/java_home is inbuilt mac utility that helps in picking the home path of right jdk based on version. 

Points to note about TCP keep Alive and basic Linux system tools to configure them at kernel level

TCP is connection oriented protocol.. i.e. there is a connection establishment phase, a transfer of data phase and finally a connection termination phase.

Connection establishment is using the 3 way handshake

Now based on timeout of connections in the system.. the connection could get closed on either side. When this happens the side that timed-out will send a FIN signal to close the connection and this responded by FIN/ACK

Now there can be 2 cases where due to network going down there can be no reply to the incoming signal. 

1) Dead peer - peer is down and request to him is not replied.. such case the sender knows peer is down only when sending a request to peer and peer does not reply, resulting in the connection time out. Also when sending request to peer who got restarted and lost the connection handshake he earlier had result in a reset request being replied to sender.

2) Network Partition: When connecting to systems through the NAT or through the firewall.. the in-memory network monitoring tools that hold the connections to the system has limitation of memory size and will evict connections it holds based on the LRU evict rule. Hence connections scoped with long timeout may get removed from NAT / Proxy in between and would require it to be resent.

KeepAlive is the concept of configuring TCP to send an extra signal that is empty and just expects and ACK from the receiver. This signal monitors if the peer server is active and stops avoiding wasteful sending of messages when the peer is dead. 

Configuring this a kernel level linux requires using either of the following two tools

1) sysclt - cmd line tool to set/get kernel variables

2) procfs - seen as /proc directory. this directory holds network config and kernel variables as files in the subtree under /proc. Modifying the files is as good as setting the kernel variables as per requirement

Key variables that play a role in tcp keep-alive are:

1) tcp_keepalive_time - idle time after payload message was sent that keepAlive message would be sent, this should ideally be less than timeout.

2) tcp_keepalive_intvl - irrespective of data sent.. the keepAlive message will be sent after this interval

3) tcp_keepalive_probes - number of probes sent for keepAlive that returned with error. This threshold number is used to define if connection to the remote service is down or not

Docker a light weight alternative to using VM's.

This blog is just to record some of my learnings while playing with it. 

On mac you can use docker only with virtual box as it needs to first create a VM on which you run docker image

for this purpose there is a simple command line utility boot2docker

To start

boot2docker start

export DOCKER_HOST as per msg -- better put it in you .bashrc

Other usages

boot2docker start/stop/ssh/status version info ip.. check them

using boot2docker help

docker command line can be used once boot2docker has started the vm

Containers and Images (The main funda)

Image is a disc image - this can be pushed to registry.hub.docker.com

conatiner is the running verision of the image. It also keeps versioning itself as you use it. 

any version of container can be discarded or converted to image by commiting

the commands to work with images and containers

docker images - list out images in the system. you may have none when you start

docker search <image-name> - search for image on docker hub eg.. docker search ubuntu

docker pull <image-name> - pull image from docker hub 

docker push <image-name> - push image  to docker hub 

docker history <image-name> - shows different version of this image (diff version) based on past commands run and not committed (Not sure to check this ?)

once image is pulled you can run the image in a container.

Commands for that are:

docker run <image-name> <command> -- eg docker run ubuntu uptime 

The above command starts a container with ubuntu image and runs uptime command - and streams the result out to the 

terminal and then stops the container.

docker ps -a #list all container that where created. 

Note: docker run command would have created a container with diff id for the command executed. 

This is listed using -a option as this container is not currently running.

The process:

the docker ps lists all process running in docker containers.

the docker ps lists all process running in docker containers.

process are running containers which are instance of image being acted upon commands

commands like uptime do nothing to the image but commands like sudo apt-get update  can alter the image. 

The above alteration does not alter the orignal image but creates diff Id i.e. container Ids which contains all the changes done.

The diff Id is generated for any command run even if it does not actual change the container image's files.

Hence docker run ubuntu uptime - generated a diff Id (i.e. container Id) which held the version of the docker image with that command run

Now since this container process started and ended right awy doing docker ps does not list anything

But docker ps -a shows all process that where ran earlier along with the exit time.

hence you can fetch run the container version of the image using docker ps -a

Now to start this version of the container of the image exclusively you can use

docker start <container_id>

This starts the container.. and executes the command that you had done i.e. uptime and automatically dies

To keep the container running and work on its shell 

docker run -ti ubuntu /bin/bash   # -t for tty session -i for interactive session /bin/bash is the command run

This gets you logged into the container which is now running without stoping and allows you to work in it like you have ssh into another vm

Now on a different terminal if you do

docker ps

you will notice the running container

You exit the running container like exiting from ssh. exit will also stop this container.

docker ps -a will show you that you have exited

docker start <container_id> of the last one you exited will restart the container with the same command /bin/bash with -i and -t

docker attach<container_id> will attach you this running container

now the old container with your change if you made any when you ran it last time will be there and you can work further on it.

Similar to git commits. You can ignore last version jump to the prior version incase you made any mistake while working with container.

This makes rollback of VM very simple.

Note: running docker run -t -i ubuntu /bin/bash -- will start a container from fresh ubuntu image and not where you last left it.

To work on old version of your changes you need to start and attach to the container.

At work there are many uses cases where I need to mine through logs to trouble shoot issues. Here I am going to share a few of my learnings. I have divided them into use cases for a pragmatic explanation.

All web logs usually follow a pattern - Assuming that the logging is through one of the logging frameworks having appenders with predefined patterns like

<timestmap>: <IP>:<HTTPMethod>:<RequestID>:<Path>:<javaClass>: message

Use Case 1:

Finding data and extracting meaningful info through split and performing operations on extracted data.

Yes grep is the wonderful tool to find what you need. But then having found those lines.. Translation of the data or data massaging is needed to be able perform other business logics on them is where I learnt the new approach with read/IFS. Guess this was the old approach before awk.

My earlier approach would be 

grep <pattern> | awk {' { <logic here>;  print " <output here>"; }'  | <business logic script >

The alternative way to do this:

#Readin delimited data - using IFS

# IFS = internal file separator

# IFS defaults to <space> <tab> <newline>

# IFS works with read to separate input to required subparts

IFS=':'

grep pattern  /tmp/delim_file | while read -r date ip requestId

do

  echo "date is $date"

  # <business logic scrip here>

done

IFS here binds with read and does the trick.This could be handy when we want to split and execute different bash commands on each split chars right away as that is not easy to do with awk.

Use case 2:

grep a number of requestIDs (400) in a file with logs. the log files are huge 20GB.

One approach to do this 

cat fileWithRequestIds.txt | while read reqId

 do 

  # -m 1 species to stop on first match

   grep -m 1 $reqId logfiles*.log 

done

But doing so took about 4 hrs to get done.  in fact when run 4 such process in parallel on a box with 8GB Ram and 4 core CPU .. I noticed the boxes load average was above 4.5. ( you can find load average using uptime)

Running top on the box revealed, that  on the all the 4 cores cpu us (user thread usage) was only 5%  sy (system thread usage) 7% but wa (wait time on disk seek) was 70%.

This made sense as log file size was huge order of 20GB and with the systems 8GB ram all the contents of file can never be in memory hence grep required disk seek to occur to load contents of file to RAM and flush RAM from the old contents that were searched.

And the above process was happening 400 times for each request Id in the requestID file hence the delay in running this process.

Now a quicker approach to solve the problem is using grep with or arguments.

grep 'reqId1\| reqid2' logs # match either reqId1 or reqId2

Note in above use case -m is not specified as we want it to go through entire file and match all reqId in the or operation.

Note: there is a limit to number size of the text string with or conditions in grep. You can use this for large number of requestIds.

The next day I ran them same logic with the new code

# build pattern for grep as explained above

grepPattern=$(cat requestIds.txt | awk '{printf $1"\|" }' } sed 's/.\{2\}$//g')

# search logs and send them to businesslogic.jar .

grep $grepPattern logs*.log | xargs  java -jar businesslogic.jar

This completed in 15 min.

Use case 3

Passing stream of data to business logic programs in other languages.

Streams are the one way of communicating between process, and different programs. At work there are times when there is some business logic built in python, some in java and the requirement is read from log file pass them to python and get the output to java.

In all languages there are two modes of reading input:

1) args - ( Java - String[] args, python - sys.argv, bash - $@ )

2) stdin -( Java - System.in, python - sys.stdin, bash - read )

when we use pipe and input or output redirection we specify stdin as the input mode, eg..

cat file | java -jar somejar.jar #main method here needs to read from System.in

I have realized that we can extract more performance by using args rather than stdin. This with the application of xargs.

xargs converts stream input to args.

cat file | xargs java -jar somejar.jar # here String[] args gets the file contents - one line per index in args[]

The performance improvement can be achieve by employing xargs options.

Consider the below case

cat file | python doSomething.py | java -jar doSomethingElse.jar 

Here are 3 process are spawned and connected by pipe  and data flow rate from one to other is limited by slowest running of the 3. If doSomething.py is slow, then both cat and doSomethingElse.jar are waiting .

Another approach using xargs

cat file | xargs -n20 -P 4 python doSomething.py | xargs -n20 -P 2 java -jar doSomethingElse.jar

Here we have 1 process for cat, 4 process of doSomething.py reading 20 lines each and 2 process of doSomethingElse.jar reading 20 lines each.

ss is alternative to netstat.. 

netstat has been deprecated in many networking tools and repalced by ss

The ss command is capable of showing more information than the netstat and is faster. The netstat command reads various /proc files to gather information. However this approach falls weak when there are lots of connections to display. This makes it slower.

The ss command gets its information directly from kernel space. The options used with the ss commands are very similar to netstat making it an easy replacement.

# get all connections

ss | less

ss -t # get tcp connections not in listen mode (server programs)

ss -u # get udp connections not in listen mode

ss -x # get unix socket pipe connections

ss -ta # get all tcp connections

ss -a -A tcp # same as above

ss -au # get all udp connections

ss -a -A udp  # same as above

ss -nt - all tcp without host name

ss -ltn - listening tcp without host resolution

ss -ltp - listening tcp with PID and name

ss -s # prints statstics

ss -tn -o # tcp connection with domain host and show keepalive timer

ss -tl4  #ip4 connections 

ss -tl -f inet

ss -tl6  #ip6 connections

ss -tl -f inet6

- Filters in ss

$ ss [ OPTIONS ] [ STATE-FILTER ] [ ADDRESS-FILTER]

ss -t4 state established

ss -t4 state time-wait

states can be:

1. established 

2. syn-sent 

3. syn-recv 

4. fin-wait-1 

5. fin-wait-2 

6. time-wait 

7. closed 

8. close-wait 

9. last-ack 

10. closing 

11. all - All of the above states 

12. connected - All the states except for listen and closed 

13. synchronized - All the connected states except for syn-sent 

14. bucket - Show states, which are maintained as minisockets, i.e. time-wait and syn-recv. 

15. big - Opposite to bucket state.

ss -at '( dport = :http )'ss -at '( dport = :80 )'

ss -at '( dst :443 or dst :80 )' # note dport = :80 same as dst :80

ss -nt dst 323.123.12.12 #Filter by address also supports address:port

ss -nt dst 323.123.12.12/16 #CIDR notation is also supported - prefix match

ss is very powerfull to filter based on source and destination ips and port

ss -nt src 127.0.0.1 sport gt :5000 # gt => greater than

#Other port compartors

le => less than or equal to

ge => greater than or equal to

eq => equal to

ne => not equal to

- combination of state filter and port filter

ss -nt state connected dport = :80 # all connections to port http service (mostly through browser)

The ps command on linux is one of the most basic commands for viewing the processes running on the system. It provides a snapshot of the current processes along with detailed information like user id, cpu usage, memory usage, command name etc. It does not display data in real time like top or htop commands. But even though being simpler in features and output it is still an essential process management/monitoring tool that every linux newbie should know about and learn well.

The PS command comes with 2 unusual syntax styles (BSD and UNIX)

ps aux (BSD)

ps -ef (UNIX) # Note - comes before args

We can mix the two styles as in ps ax -f

ps a #displays all process in random order - (I guess the most recently active/chagne process is displayed on top)

ps -e / ps ax #displays all process (note these are sorted by PID asc (starting from 1))

ps -ef / ps aux  #displays detailed info of all process (u and -f)

For all usernames if the length is greater than 8 characters then ps will fall back to show only the UID instead of username.

ps -f -u hadoop # list all process by user/uid hadoop

ps u -u haddop # same as above in BSD style

ps -fC squid #show process by name  (exact match on process name)

ps u -C squid #same as above with BSD + unix style -C

alternatively use

ps -ef | grep processName - for partial name search

note: both u and -f imply detailed info

ps -f -p 12321, 2122 # Search by process ID

ps u -p 12321, 2122 # Search by process ID

Get the most CPU intensive process (CPU bound)

 ps aux --sort=-pcpu | head -5

Similarly for memory intesive (IO bound)

 ps aux --sort=-pmem | head -5

NOTE: above output are not real time as top command

While running process like nodejs and building code that fork process - its useful 

to check the parent child relationships between process

ps -f --forest -C node

ps u --forest -C node # same as above

ps -o pid,uname,comm  -C nginx # display specific details of hte process in this case pid, user name, command

  PID USER     COMMAND

 7608 nginx    nginx

 8016 root     nginx

- above is useful to check child and parent process in above output 8016 is the parent

ps -f --ppid 8016 # returns all child process of the parent pid 8016

ps -p 8016 -L - display are threads created by this process. (separate execution context which share the same memory allocated to this process)

The output of this works better when invoked on java process (java which supports mulithreaded executions)

ps -o <FiledNames> lets you choose the columns you like to output

ps -e -o pid,comm,cmd,etime | grep java