SeaLion Blog

When SeaLion 1.0 was out, all it could do was to collect the raw output of commands executed by the agent and display those in a web interface using a beautiful timeline display. It was more of an activity recorder back then than a complete monitoring solution. Plotting this recorded activity outputs using a time series data visualisation was the next step in the journey towards becoming a full-fledged monitoring tool. We had this target of squeezing maximum information in the desktop screen so that a user can have a view of hundreds of servers and pinpoint the problematic servers at one look. From that point onwards the search for a visualisation tool which could serve to the set of our requirements had started.

We evaluated a handful of libraries and frameworks, but we hit certain limitations of those. For instance, Line charts gave a good view of spikes but displaying more than three graphs(one each for a metric) for a server, cluttered the graphs so much that it almost made no sense visually. Heat maps, on the other hand, had beautiful rendering, but there was no way to depict comparison among two servers, the color density being an absolute value. Also, we had a limitation on the height a graph can attain, maximum value on the y-axis to be precise. Since higher values meant more vertical space and the maximum values were unpredictable, it was harder for us to modify the graphs to fit our needs. Hence, it was clear that line charts, column charts or heat maps were not the way to go.

A lot of research and prototypes later, we stumbled upon horizon charts, suggested by Sabyasachi Ruj, one of the senior engineers of our organisation. Horizon chart was developed keeping the space utilization in mind. It uses colors to depict the higher values and the direction of offsets to show negatives. Horizon charts not only reduced the screen space required but also provided a better visualization regarding spotting a huge change in the trend. Let’s dig a little deep into how horizon charts work.

The plotting mechanism for horizon graph is similar to regular column charts except that the height does not serve as the quantity measurement parameter. A regular column chart displays information such that the peaks pertain to higher values. For example, the following graph shows a trend where it is easy to spot a spike.

But you can also conclude that the height of chart would keep increasing with the magnitude of the values. Here is how horizon graphs solve it. It divides the graph horizontally into two parts and paints them in different colors keeping darker color at the higher position as you can see in the following picture.

Then it stacks dark colored partition over the lighter one.

Here is how the color density does the math. If there is only one color and a set of values from 1 to 100,  the height of peak decides the higher number. When there are two colors, the lighter color shows values from 1 to 50, and the darker one shows 51 to 100. Hence, darker the color, higher the value and the height will be reduced by a factor of the number of colors used. You can find a further detailed explanation here.

Since there is no need of the y-axis in these plots and the color densities are relative, we got an easy solution to our problem of comparison among different graphs. One can easily focus on the dark colors and get to the cause of unusual data trends.

Horizon charts, though difficult to process from the start helped us solve a great deal. Not only did we create beautiful data visualization but we also succeeded in rendering a lot of information on a limited screen space.

Bolstering the growth of your organization requires a tremendous amount of enhancement to your systems and processes. Building your IT systems, servers and back end can turn out to be the make or break point for your organization. Scaling up in the right manner will decide the future growth path and sustainability.

If you're a Linux user, it comes with its own set of challenges. People have been known to be stuck in loop managing the Linux servers. Today where all startups, small & large corporations are scouting for a dependable and stable environment, we have undertaken the enormous responsibility of providing the best solution for a Linux server administrator. SeaLion ensures you save time and increase productivity while managing your Linux servers. All your issues and challenges will be covered under the product. Making sure that you don't have to waste time when something goes wrong.

After extensive testing, we have come up with the perfect balance of raw data and charting helping you make critical decisions without distracting you with unwanted information and save the day. Manual checking for information consumes extensive time and makes you vulnerable to make mistakes. SeaLion covers all the critical pain points of segregating raw outputs and helps you take over critical decisions.

SeaLion would automate the entire process of monitoring your Linux server with raw data you need and notify you for all custom and critical events. Thus, we ensure you save your precious time and downtime. Installing SeaLion is an easy process and takes only 5 minutes of your time. Just one curl command and you are set. SeaLion allows you to cut through the noise and view the raw output for critical events across servers with one single tap.

Custom Metrics:

We integrated Custom Metrics into SeaLion after diligent testing to find relevant metrics. Knowing that all information is not important, we gave you the control to modify and get the information you need. Customizing the experience and handing you the correct data at the appropriate time. You can try the whole SeaLion experience here.

Done with Sealion agent installation on your Linux server? SeaLion comes bundled with a bucket of predefined commands and services like Apache, Nginx, MySQL, MongoDB and Redis. A command is a full-fledged bash script which writes to stdout. You can get more details about commands here. These commands can be added by looking at the usability and user recommendation. Integrated commands are useful in the majority of cases however you might want to add your own command to monitor. The outputs produced by the commands come in handy when debugging an issue. You can also create metrics to extract values from the output produced and set up alerts as well as plot graphs. So the next thing that pops up in your mind is how do I add a command to monitor?

To add a command, just navigate to the Settings page and click on the ‘Add command’ option under the Commands section.

Let us take an example of lsof and try adding that in SeaLion. The command ‘lsof’ basically lists the open files and process that opened them.

Output of ‘lsof’ looks something like this

COMMAND  PID  TID  USER  FD  TYPE   DEVICE  SIZE/OFF   NODE        NAME

init                 1              root   cwd    DIR      8,1                            4096           2 /

upstart-u       458         root   cwd    DIR      8,1                            4096            2 /

rpcbind         610         root     6u    IPv4    7924                            0t0         UDP *:sunrpc

rpc.statd       685         statd     4u    IPv4   8075                           0t0        UDP localhost:861

dhclient        721          root      5u   IPv4   8079                            0t0         UDP *:bootpc

upstart-f       945         root   cwd    DIR      8,1                             4096           2 /

nginx           1322         root    10u    IPv4   10310                          0t0        TCP *:http (LISTEN)

In this command, it might be useful to filter all the open files and processes by a particular user or type. If you wish to monitor all the open files and processes by ‘root’ or maybe you want to list all files with type ‘IP’. So the command becomes:

lsof | awk "NR == 1 || \$3 ~ /root/ {print \$0}"

The output of the same command would be:

COMMAND    PID  TID USER  FD    TYPE   DEVICE SIZE/OFF   NODE  NAME

init                    1            root  cwd   DIR        8,1                             4096       2 /

upstart-u       458          root  cwd   DIR        8,1                              4096       2 /

rpcbind         610           root  6u    IPv4      7924                              0t0     UDP *:sunrpc

dhclient        721          root   5u    IPv4       8079                             0t0     UDP *:bootpc

upstart-f      945           root  cwd   DIR       8,1                                4096       2 /

nginx          1322          root  10u   IPv4     10310                             0t0     TCP *:http (LISTEN)

You can monitor this command by adding it to the SeaLion interface.

The commands might have sensitive information such as your username, password, etc. In such cases, you need to hide the information to prevent the misuse of data. SeaLion protects this information by using the concept of ‘environment variables’ in your script.

To illustrate the use of environment variables let’s take the example of ‘lsof’ command mentioned earlier. The command is hardcoded to find all open files and processes by ‘root.' This might not serve your purpose in the case where you want to monitor all the processes by a different user, or maybe you want to list the processes by a different category. Apart from hiding the sensitive information, environment variables might also be useful to match an existing field with a variable name. Using the environment variables instead of hard-coded values will change the command as follows:

lsof | awk "NR == 1 || \$ $SL_FIELD_INDEX ~/$SL_FIELD_REGEX/ {print \$0}"

SeaLion follows this convention of adding an environment variable to a command by starting the environment variable name with “SL_” followed by at least one digit or uppercase letters. This requires you to set up two environment variables. One will store the field you want to match the variable with, and other will store the variable value. We can name them as SL_FIELD_INDEX and SL_FIELD_REGEX respectively. You can read more about environment variables here.

Specify the time interval as per your needs. The command will execute periodically in that time interval. It is ‘5’ minutes by default.

Next interface asks you to set-up the value for environment variables SL_FIELD_INDEX, SL_FIELD_REGEX. Provide the respective values. Alternatively, you can do it by adding the value of environment variables in config.JSON file by selecting ‘Configure in host’ option or set it later by disabling them.

A command is executed at category level hence the last step is selecting the desired category. By default, the command will be scheduled in all the categories.

After Scheduling the command, you can view the output of the command in your workspace.

To conclude, you can now add and monitor any command on SeaLion, so you can always keep an eye on your Linux servers.

NOTE: As you can see from the output, the command ‘lsof’ returns ‘permission denied’ with most of the processes because SeaLion agent runs as an unprivileged user (SeaLion) under SeaLion group. You can execute the command as a superuser by adding “sudo” in front and editing the sudoers.d file. You also need to prefix sudo to the command while scheduling it from SeaLion web interface. This can be done by following this link.

This blog post is written by Bhuvan Kapoor  of Team SeaLion.

Finding it difficult to understand how custom metrics work in SeaLion?

Well, nothing to worry about as this 5-minute tutorial will help you get through. A custom metric is a framework that allows to parse value from the raw output of a command for consistent monitoring. SeaLion had already been providing the flexibility to add your own commands and record their outputs. A command is a bash script which writes to stdout and it can be added through the interface by navigating to settings accessible at the top right corner of the dashboard. You can read more about commands here. Once you are done adding a command, you can easily create a metric on top of it.

Let’s consider the example of a command, disk usage, obtained by running df -h and dive deep into the steps of creating a custom metric.

So, the output produced by df -h looks like this:

Filesystem   Size   Used   Avail   Use%    Mounted on

/dev/xvda1  40G     3.2G    37G     9%         /

tmpfs          829M       0     829M    0%        /dev/shm

/dev/md127 40G    424M   40G     2%        /srv/mongodb

This output shows the percentage use of all the partitions with respective mount points. Suppose you are interested to know the usage for the /dev/md127 mounted on /srv/mongodb as you might need to resize the volume as it grows. Keeping a check on this will require you to browse through the command output every now and then. To save you from this monotonous activity, custom metrics come to the rescue. With the help of custom metrics framework, you can easily extract out the disk usage of MongoDB volume and can draw its graph to keep a tab on the trend. Not only this, you can create an alert on this metric which can notify you if the usage crosses the specified limit.  

Here is how the custom metric on disk usage can be created to extract the MongoDB volume use percentage. As the first step, you need to navigate to metric settings and select the command df -h.

Clicking on ‘Add Metric’ will open up a pop-up.

Now, let us understand each field one by one.

Name: The identification of custom metric. We are going to name our metric as MongoDB volume.

Type: A metric can either be cumulative or non-cumulative. If the output produced by a command is cumulative in nature, then effective metric value is obtained by calculating the difference between the value from current collection and the previous collection. On the other hand, non-cumulative values can be understood as absolute values at any given time. MongoDB volume in discussion here is a noncumulative metric.

Unit: A metric may or may not have a unit symbol but if it does, you can provide it while adding the metric. In this case, the unit will be %. Also, if a metric has factorizable unit symbols like seconds, minutes, hours etc you can easily configure those symbols with respective factors. Please refer the screenshot here for better understanding.

Bounds: As the name suggests, bounds is a pair of minimum and maximum values a metric can attain. In the above-mentioned example, 0 will be the minimum bound and 100 will be the maximum as the value is in percentage. The bounds provided by the user saves us the approximation and helps us to draw accurate graphs.

Code: For metric to be parsed you just need to write a simple parsing code in Python. The parser needs to process the raw output stored in the global variable command_output and store the result in metric_value.The following parser can be written for extracting MongoDB volume.

As you can see the code loops through the output lines looking for /srv/mongodb mount point and then stores the respective use percentage in metric_value. Refer this knowledge base article for more information on the dos and don’ts of writing a parser.

Click Save and the custom metric MongoDB Volume is ready.

You can now move to dashboard to see the graph plotted for MongoDB Volume. Also, you can set alerts on this metric which can warn you about the metric hitting the threshold.

Also, if you need to stop the monitoring the metric, you can do so by disabling it using the small green control depicted in the picture. It will stop the metric data from being collected temporarily. However, you can enable it again as you need it.

In conclusion, this custom parsing framework allows you to extract any value from the output of any command that you need to monitor be it MongoDB volume from disk usage or max_used_connections of MySQL. This simple yet powerful framework can help you attain enormous monitoring power at one platform.  

This blog post is written by Yogita Sharma of Team SeaLion.

We are excited to announce the release of the most powerful Sealion v3.0. Our development team has been working against the clock to make user’s life easy by integrating highly productive and powerful features in SeaLion.

SeaLion now supports custom metrics which can be used to create, schedule and monitor custom metrics in horizon charts - we just took server monitoring to whole another level!

SeaLion has been built grounds up to monitor hundreds of servers, and we have increased the bandwidth of the product regarding service compatibility by supporting new services such as MongoDB, Redis, NGINX, Apache, and MySQL. One agent to monitor all services and use the custom framework to monitor any other Linux based services.

SeaLion is a Linux Server Monitoring Tool, which allows you to cut through the noise and view the raw output for critical events across servers with one single tap.

Unlike all commercial solutions, SeaLion perfectly integrates the right amount of charting and the raw output of commands so you can take immediate actions without having to absorb processed data for business intelligence, unwanted insights, etc. It's literally like having Linux terminal open across servers (identical to what you used to do during college days) with classic UI and appropriate charting to identify the cause of performance degradation.

Try the new version

Why top reports a different value for CPU usage compared to SeaLion?

This is one of the questions that SeaLion team is asked every now and then. It is because most users schedule top command as top -b -n1 where the option -n specifies number of iterations over which top should summarize. But to summarize effective CPU utilization you should have at least two iterations as -n1 shows the total CPU utilization since the system boot. To do so, one can use -n2 which will let top sample the output over two iterations. But that’s not it, one should know that this sampling is done over a default time interval of 3 seconds or the value configured in toprc which means it gives effective utilization over a period of 3 seconds by default. However, this value could be given explicitly using the option -d in top command. So, running top -b -n2 -d1 will give you the effective CPU utilization, sampling the contents of two iterations over a period of 1 second.

Like top, SeaLion agent also samples contents of /proc/stat, at an interval of 1 second. So, it is due to this difference of sample intervals and missing out on correct number of iterations that there is no correspondence between the values reported by top and SeaLion.

To confirm the same, we ran an experimental Python script which accepts two arguments.The argument -p  specifies the sample interval for /proc/stat and -t specifies the sample interval for top command. The script basically starts two subprocesses in parallel. One of them calculates directly from /proc/stat and the other one uses the output from top. It also creates multiple threads to simulate CPU usage. The script is run with Jython as it has no GIL which allows it to take full advantage of multithreading to simulate CPU usage. One can also use option -s to see details of process in top output. The script can be found here.

Let’s run the script by specifying the sample interval only for /proc/stat.

As you can see, CPU utilization reported by /proc/stat and that reported by top does not match. This happened because we didn’t specify the interval for top, it runs with the default sampling interval of 3 seconds. But in case of /proc/stat it was specified to run with sampling interval of 1 second. The time taken by script which is 3.169 seconds verifies that top command took around 3 seconds to finish.

Let’s see what happens when the intervals are same.

In this case utilization reported by /proc/stat and top is correspondent. As it is clear from the screenshot, top command runs with option -d using the argument provided to the script thereby using it as sampling interval. Since, both commands use the same interval, the outputs are correspondent.

Why some processes show CPU usage more than 100% in top output?

CPU usage for some processes, as reported by top, sometimes shoots higher than 100%. It happens only when a multithreaded process is running on a machine with multiple processors/cores. Effective CPU utilization for a process is calculated as a percentage of number of ticks elapsed by CPU being in user mode or kernel mode to the total number of ticks elapsed. If it is a multithreaded process, other cores of processor are also utilized summing the total utilization percentage to be more than 100.

To verify the same, the above mentioned script was executed on a dual core machine using option -s, showing the effective CPU utilization for a process as reported by top output.

In this case 458 is the total number of ticks for the process in user mode and kernel mode combined and that was over 3 seconds. Since 1 tick equals 10 ms, so 458 ticks equals 4.58 seconds and calculating percentage as 4.58/3 * 100 will give you 152.67, which is almost equal to the value reported by top.

So, how do I find actual CPU usage?

CPU usage refers to the time taken by CPU cycles when it is in user mode or kernel mode. Niced processes also belong to user mode whereas software and hardware interrupts are handled in kernel mode. If it is not doing any of these then either the CPU is idle or it is waiting for an I/O to complete.

From the above results, effective utilization can be calculated as sum of %us, %sy, %ni, %hi and %si, which in this case is 68.71%. Although some people may question as to why iowait is not included in CPU usage when very high value of iowait could be a reason for bad performance. It is because CPU usage by definition includes only the time spent by CPU cycles in running some processes or handling interrupts. When any process is waiting for an I/O to complete, it is not contributing to CPU cycles. So, ideally iowait is considered as a subset of idle CPU but it is true that high iowait is a bad sign and one needs to be aware of it. If a process is in uninterruptible sleep doing a disk read/write, it will result in high iowait and in that case Load Average will start spiking. However, if a process is in interruptible sleep for example waiting for a socket connection, it will contribute to high iowait but that will occur in small bursts. Since the situation of processes in uninterruptible sleep is not something which will frequently occur, SeaLion does not include iowait in CPU usage while calculating effective CPU utilization.

This blog post is written by Yogita Sharma of Team SeaLion.

While SeaLion is being embraced by SysAdmins and DevOps around the world, we’re constantly working on improving the experience. We have now added graphs to the workspace making it easier to investigate trends and view raw output simultaneously.

You can now easily glance through the graph in the workspace to quickly spot a trend/issue. Select a point in the graph to view the raw output without leaving the workspace. This feature makes it easier to correlate trends in the graphs to actual data to help you debug faster.

Other enhancements

The formatting of the raw output timeline also has been improved for better readability. The command filter in the workspace is now collapsible to make better use of the screen real estate.

The v2.2 update has already been rolled out to all SeaLion users, so log in and experience the new improvements in the workspace.

Check the latest update

New to SeaLion? Signup

While we continue to work on new features and improvements, it’ll be great to get your feedback for this update.

Introducing major updates to your favorite linux server monitoring tool. With a fresh design and important new features, we have improved the overall SeaLion experience. Here are the details:

A swanky new Dashboard and workspace

We had to rethink each pixel to clear out all the distractions so you can focus on what matters the most: your data. But, there's a lot more to it than a new look. Here are the not-so-subtle changes we've made to your dashboard and workspace:

Adaptive graphs 

SeaLion was built for users who need to monitor hundreds of categories and servers from a single dashboard. However, some of you have 2-3 categories to monitor, resulting in lot of empty space. The new adaptive layout now magnifies your graphs, utilizing all screen real estate available. Ensuring you get the best view & don’t miss out on any of the details.

Collapsible output view

We said goodbye to the ugly checkboxes, making it easier to add multiple servers and view their output. The output view is collapsed by default. You can view the output and minimize it if you wish to check the graph trend or add more servers to output view.

Multiple metrics at once

You could always switch between various metrics like CPU, Load Average, Memory Usage before. Now you can view multiple metrics at once across all your categories and servers. Making it easier for you to spot a trend and debug quickly.

Jump timelines

A new option that enables you to quickly jump back and forth in your timeline to see the state of your servers with just a single click.

Quickly remove servers from output

While viewing output, you can now remove servers, without returning to your SeaLion Dashboard.

The v2.1 update has already been rolled out to all SeaLion users, so log in and experience the new dashboard and workspace!

Check the latest update

New to SeaLion? Signup

While we continue to work on bringing new features and improvements, it’ll be great to get your feedback for this update.

A system administrator experiences a wide array of emotions in his day-to-day job. Things keep breaking, we keep fixing them. There is nothing called a normal day for a SysAdmin. Each day is full of surprises, challenges, frustration and joy. It's hard to explain these emotions in words. We’ve tried to compile a list of GIFs which every SysAdmin will be able to relate to.

Sudo

When a user asks for root access

Handling ticket escalation

Deploying a cluster

Trying something new in production

Being told about security by co-workers

Keeping systems running

Priority inversion

When the load on your system goes high

When someone tells you "that thing isn't working again"

Working without the right tools

When a new admin suggests you to rewrite your tools in a new language

Dodging DDOS attacks is just another task for you

We are delighted to introduce you to SeaLion 2, today. With refreshing new design, charts, alerts and tons of improvements, SeaLion 2 is our biggest update yet. Before we get to what's new in detail, let me recap our journey so far. SeaLion was a tool created for our internal needs. We sensed a demand for such a product and launched it for the public in the fall of 2013. Soon we had a good number of people using us. Some loved what we did, some didn't; feedback started pouring in; people continued to use us and users grew inspite of hiccups. We were now sure that we were building something that people care to use. We took note of every little thing a user has to say. We continued to iterate. And today, here we are with SeaLion 2. It is the manifestation of a year long interaction and feedback cycles with hundreds of system admins.

What's new?

Dashboard

Introducing to your new workspace - the dashboard. It lets you visualize metrics from all your servers in one page. A bird's eye view of all your servers. We chose horizon charts to plot the server metrics. These charts make better use of vertical space than standard area charts, allowing you to see many more metrics at-a-glance and thereby increases the likelihood of discovering an anomaly. See something odd? just click on that datapoint to jump directly to the command outputs of that period. Want to compare metrics of different server? just mouse over on a datapoint and metrics from all servers for that timestamp are highlighted.

Alerts

SeaLion has always helped you in troubleshooting servers. However, you had to use it in conjunction with a tool/script to alert you when things go wrong. Not anymore. SeaLion 2 sends you instant email alerts when the server vitals cross thresholds. Plus, the email alerts will take you directly to the timeline of the server and snap to the time period where the metrics would have crossed the thresholds. Zeroing in on an issue has never been easier. With SeaLion 2 you also get the summary of server health in your inbox as a daily digest.

Pricing

We will be charging a small fee for SeaLion starting from today. Monetizing a product not only helps the team with cash flow, but also boosts the confidence of serious users. Trust me, many users turned down SeaLion because they were skeptical about our intentions. It's natural for one to suspect when good things come for free :-) BTW, we are still giving away 2 servers for free so that one can give us a spin and buy a subscription as they grow.

Agent goes open source

SeaLion service was down from 09:00:00 UTC to 13:30:00 UTC on Dec 14th. Result of which, for few of you, the SeaLion agents monitoring your servers crashed. Those affected will be missing server output in their SeaLion dashboards, starting Dec 14th, 09:00:00. We are really sorry about that. We have worked round the clock over the weekend and fixed issues. Please restart your SeaLion agents to start seeing server outputs again. Here’s the command: sudo service sealion restart

What went wrong?

Node.js and socket.io are part of our server side code stack. One of our node.js instance hanged (we are investigating the reasons) and SeaLion service went down. We restarted the node.js instance in discussion and the service was restored. It did not end there. One thing led to another. All socket.io connections that were established with the non-responsive node.js instance timed out. We had missed to handle the exception thrown on connection timeout. This brought the SeaLion agents down. Hence, no output was collected from the affected servers since the outage.

What did we fix?

We launched our product on Hacker News and managed to get to the top 3 posts and stayed on the frontpage for more than 24 hours. It was no serendipity, but planned execution. We got great feedback and a good number of users. Here’s how we did it.

Tell a story

First figure out what do you want from the post, right? No, you are wrong. Get into the shoes of an avid HN reader and think what she wants, and align your post to her taste. She doesn’t want to read one more product press release or marketing BS. The basics:

Hacker News is a social news website that caters to programmers and entrepreneurs, delivering content related to computer science and entrepreneurship. - Wikipedia

The readers are here to learn. So, teach what you know. That’s what we did. We shared the story behind our product. What inspired us to create a product of that kind? Why we are the right guys to do it? What are the possible directions we could take the product development in to? It was a programmer talking to a programmer in her jargon. We told a story that an HN reader yearns to listen.

Get the post title right

An HN post title is of 80 characters. That’s the space you have to lure the reader. We spent more time coming up with the title, than to write the product launch blog. We sat down and listed all the posts that did well in the past few months. Later, we narrowed down on the ones that made sense to us and grouped them by the tone. Here are those:

a. Straightforward titles about the product

Show HN: Meteor, a realtime JavaScript framework Show HN: Automatic weekly meal planner, also plans your leftovers Show HN: Ditch Black Text to Read Faster, Easier Show HN: Codecademy.com, the easiest way to learn to code Show HN: a web-based xkcd-style comix editor Show HN: Remove backgrounds from images online Show HN: Imagine a search engine that removed top million sites from its index Show HN: movies.io — torrent search like it should be Show HN: Hackershelf. Community curated collection of legally free books Show HN: Webflow - Design responsive websites visually Show HN: fontBomb - Stylishly destroy the web Show HN: Non-Confusing, Visually Correct Slider Toggle UI

b. Problem/Solution title

Show HN: We got tired of asking 'What browser are you using?' and created this

c. Title by an individual

Show HN: 6 months ago I quit my job to pursue indie dev. Here is our first game Show HN: Weekend project, Intro.js Show HN: I Created the Inverse of Angry Birds Show HN: My seven minute workout timer evening project Show HN: I hacked my microwave with a Raspberry Pi

d. Title by a group

Show HN: We open sourced Lockitron's crowdfunding app Show HN: We just built a site that saves lives We then came up with the titles that we could use for our product Show HN: Hosted Linux sar - SeaLion Show HN: We got tired of asking 'Why Linux server went belly up' & created this Show HN: We’re tired of asking 'What’s wrong with our Linux servers' & created this Show HN: Tired of asking "What's wrong with our Linux servers?", we built this Show HN: The Linux sys-admin story. Why did we create SeaLion? Show HN: Why did we create SeaLion? The Linux sys-admin story Show HN: A tool from sys-admins to sys-admins. The SeaLion story. Show HN: We wanted to simplify the tasks of a sys admin and created this Show HN: Dear god, save sys-admins. The SeaLion story. Show HN: A tool from sysadmins to sysadmins. Hosted Linux SAR. Show HN: Hosted Linux SAR. A tool from sysadmins to sysadmins. Our product is a sysadmin tool. So we ran these titles by resident sysadmins on what would push their buttons and chose this: Show HN: Tired of asking "What's wrong with our Linux servers?", we built this

This title did make the sysadmins tick. We got nearly 10K visits in a few hours and brought down our servers. I’m saving the discussion on how our servers went down and how we fixed them for some other day.

The HN moderators soon realized that the title is juicy and edited it out. One more point to prove that title was indeed juicy and it was working.

Engage

Once, the post is on front page, you start getting comments, too many of them. More than one person can handle. The earlier you reply, the better chances of the post staying in front page. It’s a no brainer. We quickly asked our sysadmins around for help. There was team of 4 working on a quality reply for each comment. HNers love engaging.

Good Karma

Good Karma helps; both in life and on Hacker News. And, it takes time and doggedness to gain good Karma. We have spent around a year to build our HN Karma.

Crack the post time

Timing is key. We submitted at 8 AM EST. The idea was to choose a time when the noise is less and the HNers are waking up. When they hit their newsreaders they would be seeing us first. The chances of crawling up to frontpage is high when the age of the post is less. And, also look for that perfect day when there's no major tech event lined up. The last thing you would want is to time your launch with a big Apple event and risk getting lost in the noise created by the big guys. Techmeme's event list comes very handy here.

No spam, please

What is SeaLion?

SeaLion is a Linux system activity recorder, built from the ground up for troubleshooting hundreds of servers.

Why did we create SeaLion?

This is the first post from the SeaLion team and I guess it’s the right place to answer this question. We’ve another product line - CloudMagic which is an award winning email app. We manage petabytes of indexed user data spread across hundreds of servers. This data is hosted on AWS. Being a search engine, the search speed is critical. We take pride in being known for blazing fast search speeds. When a search query takes more than the threshold value, we get an alert and we debug. Typically, it’s important at this point to determine if the issue was with the hosting provider or in our code. Inspite of having a suite of monitoring tools, we always fall back to the output of traditional commands. What took time? We check the usual suspects - CPU, Disk IO, Network, Memory etc. Where? Sift through the layers of database server, web server and application server. We first zero in on the server we suspect, SSH into it. Look for the log files --you know how log rotation makes it tough to look for a file-- and fish out the relevant log file. Uncompress it. grep, sort to jump to the specific time. If everything is fine, investigate the output of next set of commands. Rinse and repeat for all commands till we find the issue. Ah! This is indeed cumbersome and time consuming. Imagine doing it every time you get alerts for various other parameters.

Reminds one of your debugging terminal? And that’s why we built SeaLion. Troubleshooting across a large number of servers is broken. With SeaLion, we now just open the browser, and the output of standard commands is available in a nice tabbed interface. We can now jump to a specific period in time with a single click. One can add their own set of commands too. We can also compare the output of different servers side by side, to find out why one server is performing worse than the other of the same type. The task that used to take several minutes now gets done under a minute. I’d like to emphasize that we consciously decided not to process the output, because as said before, we system admins anyway fall back to the raw output of commands. Our eyes are accustomed to look at the output of commands and use our judgement to fix things rather than trusting shiny charts. In fact, our experience says that processing of command output masks critical information. Raw data FTW. A picture is worth a thousand words. A video? Check out this 2 minute video.

What's next for SeaLion?