Botmetric Blog

Are your cloud costs spiralling out of control?

AWS Reserved Instances can help reduce your AWS infrastructure costs.There are many important changes with respect to the new Reserved Instances, so we have put together the top insights to help you understand and better plan your RI purchases.

Image source

Save your money with Reserved Instances now! If you are new to reserving instances on Amazon Web Services then you can refer to our step by step instruction here. While reserving EC2 instances you should consider the below mentioned important key-points.

24 Hours of Billing - All RI instances will attract 24 hours of billing irrespective of utilised hours, unlike previous generation Low or Medium utilisation instances.

RI Cost vs On-demand Cost - The billing data for RI vs On-demand is computed at each billing hour for each of the instances in each availability zone. This is important to understand as if you have any weekend shut down activity for Dev/QA machines.

Break Even Periods for M3 Families - The 1-YR RI requires 7.5 months and 3-YR requires 14.5 months for break even period. So unless utilisation is more than 80%, it doesn’t make sense to do RI for 1 year.

Partial vs All Upfront Payments - The difference between Partial and All Upfront is approximately 50% for initial payment. It doesn’t make sense unless you run a pre-approved budget for Cloud where money can be spent early-on.

Partial vs All Upfront Savings - The difference between savings rate for Partial Upfront vs All Upfront is around 1% for 1 year term and maximum of 2% for 3-year period. If you are cashflow conscious then Partial Upfront makes sense and All Upfront provides ease of budgeting.

Importance of Instance Families - Choose new generation instance families like M3, C4 and R3 for your workloads. It makes sense to choose few types of instance families, so you can roll up and roll down the machine types based on future usage.

Within Botmetric's cost analytics we have a separate section dedicated on helping you do Reserved Instance planning. We are releasing an upgraded version of the RI planner along with recommendations and improved utilisation information. You can check it out here after logging into your Botmetric application.

Want all your cloud spend reports on table everyday? You are just a click away!

Botmetric helps you run a comprehensive audit of AWS workloads and get proactive insights on disaster recovery, security, cost analysis and cloud performance. You can get a daily summary of your most important audit issues. You can also track and predict your cloud spend at a granular level and optimize cost by acting upon smart recommendations.

Wondering what’s new?

We are excited to release an updated version of Botmetric, along with feature improvements and bug fixes. Feature improvements include:

1.    Enhanced user management:  A Botmetric admin can now restrict Botmetric user’s access to a limited set of AWS accounts. That is, an admin can give specific AWS account level access for users by selecting the accounts from the drop down while adding a new user as shown in the screenshot below. This feature will help if it is desired that all users should not have access to all the AWS accounts configured in Botmetric.

2.    Detailed audit preferences: Botmetric provides audit for all your cloud resources in terms of performance, DR (Disaster Recovery), security and cost. Users can now define audit preferences in greater details by using audit severity. Users are allowed to set the audit preference by selecting the type of severity such as critical, warn and info as per their requirements. This feature helps in creating custom preference set as per the user’s need.

3. New filters in cloud insights: New filters have been added to audit results. Users can now filter the audit results by selecting specific region and severity as shown in the screenshot below. In addition, users can load more than last 5 audits in the audit history section. 

We have also made a few enhancements in Performance audit and Security audit. Try out the two new audits under security audit:

-  SSL Expiry Audit

-  Domain Expiry Audit

Already a user? Login to Botmetric and check out our feature improvements on control panel and cloud insights.

We’d also love to hear your feedback or suggestions and help us provide you a better experience.

Most of us are using AWS Cloud for developing and deploying web applications with a huge concern on the performance. If it is bothering you all the time, fix it now! Here are the top 5 simple tips for improving your web applications performance and scalability. 

[Image Credit - Source]

A good performance is critical to ensure better user experience because faster performance leads to more user engagement on your web applications.

Elastic Load Balancer Level SSL Termination -  Many web applications use SSL for securely transmitting information from browser to web servers. However, SSL requires multiple handshakes, content encryption/decryption for each request. So, it would put an additional load on your web servers. If your web application is using SSL and deployed on AWS cloud then we recommend you to terminate SSL at Elastic Load Balancer (ELB) layer instead of your web/application servers. This will reduce the CPU load on your web servers and improve application response times.

Enable Compression For Static Resources  - Whether you are serving static content like images, JS, CSS or JSON/text files via CloudFront CDN or web servers, it's recommended to ensure that static content compression is enabled. With content compression on server side, the static file sizes will be reduced to improve the download performance and help you save on bandwidth costs as well.

Enable Browser Caching Headers - Modern browsers support client side caching so web applications can cache static resources (header images, logos, style sheets, javascripts etc) on browser side without downloading them from servers on every request. We recommend you to enable caching headers for all your static content whether it’s served via CloudFront or backend web servers. With client side caching headers, your web application can use browser side caching and reduce the load on backend infrastructure to improve application performance.

Leverage Browser Parallelism - Most of the modern web browsers can make 3 to 6 parallel connections to a particular domain at a given point. If your application uses many static resources like images, files, styles etc, then it’s recommended to use multiple domains like images1.myappdomain.com, images2.myappdomain.com, static.myappdomain.com to parallelise browser download of resources. All these domain can point to the same backend infrastructure like CloudFront distribution or ELB but browsers will essentially make more parallel connections to fetch resources as you are using multiple domains aliases for static resources and make your web application load faster.

Performance Review Your Web Application - You should use tools like YSlow, PageSpeed, DynaTrace etc to analyse your web application performance and implement the recommended best practices. Many web application optimisation techniques are easy to implement and will help you in achieving a better user experience.

Hope this article helps, if you are actively using AWS Cloud infrastructure then check out Botmetric to improve your operational efficiency and simplify day to day operational tasks. Botmetric also helps you perform performance audit of AWS ELB and AWS RDS which are often the main source of web application's latency issues.

If you need any help give us a shout here or reach out to our team at [email protected].

[Botmetric is pleased to kickoff the 'SmartOps' series of articles for our readers with intent to share best practices and approaches in cloud operational automation. We will share experiences, scars and good vibrations on achieving zen in the ever-challenging yet critical world of cloud operations.]

           “Doing less is more; However, there is more to do, not less;”

                                                                       - Hint in a puzzlegame

AWS resources such as EC2, EBS are extensively used for running web applications. These applications generate application logs that are important to retain yet these logs end up hogging AWS disk volumes creating performance side effects and challenges for cloud ops.  First, let’s examine the side effects and headaches that filled up disk volumes can pose and then explore how intelligent cloud automation via Botmetric can save make this activity efficient.

InAWS, EBS volumes are attached to EC2 instances as way to store data from logs and other sources. Just as human arteries get clogged due to plaque and thus impact blood flow, these storage volumes can get filled up quickly leading to system level performance issues. Some of the reasons why Cloud Ops and DevOps keep getting challenged by full disk volumes are:

System Performance Impact- Application servers with nearly full disk volumes may experience system level and possible performance level impact because the application server serving the request may run of disk space.

Data Loss- Full disk volume means risk of not being able to save all data since there may be no space to store logs.

Alert Fatigue for Ops- Full or nearly full disk volumes also signal high disk utilization. They could result in higher number of operational alerts for Cloud Ops resulting in alert fatigue. Imagine getting 50+ alerts per day on disk volumes!

Cost implications- Disk space on AWS is not free.

The impact of disk volumes getting filled up with logs is magnified much more in larger AWS deployments where hundreds of EC2 Instances are involved. Such scenarios could easily push Cloud Ops into a reactive mode, which ends up becoming counter productive.

While logs contribute to disk volumes getting full, log retention and archival is equally important for a host of reasons such as diagnostics, compliance, analytics and more.  How do you address the challenge of continually ensuring that disks are efficient and logs are not lost or deleted? Lets explore some of the options:

1. Custom Scripts - Somebody in DevOps could write or leverage custom software scripts that could move help move log files from disk (EBS) volumes on AWS to S3 archival.

2. Manual - User could manually move log files from EBS to S3 or alternative location. Both these approaches are workarounds at best because a) Custom scripts are not designed for permanence or flexibility b) Manual approaches do not scale- imagine if you had disk volumes attached to hundreds of EC2 instances c) How do make your scripts intelligent?

 3. Continuous Archival via Intelligent Automation - One other alternative is, investing in an intelligent cloud services management platform like Botmetric. Think of Botmetric as virtual Cloud Ops engineer in your team that provides smart diagnostics and operational task automation on AWS.

Here is how Botmetric could add Intelligence to Log Archival:

Motion detection /Policy based automation- Botmetric provides a intelligent policy engine that works similar to the way modern traffic light signals work-  “Sense the load, auto-initiate action, visual notification, repeat process.”

Threshold automation- User can set thresholds on Botmetric. Say if threshold for disk utilization is set at 70%, Botmetric will take action on that threshold to move log files from disk volume to S3 buckets on AWS.

Usage based automation- Only move most recently used files instead of moving all log files en masse.

Botmetric completes the loop by sending a notification to user.

Scheduled automation- Schedule a time of the day for log files to be moved.

Continuous learning- Learn trends in log creation and enforce policies for log archival based on the leanings.

Benefits of Botmetric automation based approach:

 Cloud Ops becomes proactive instead of reacting to full disk volumes.

 Clear ROI- More efficiency gained, time saved and cost savings  achieved through automation. You essentially get a virtual engineer to  do the work.

 Data Retention- No need to expunge or randomly delete logs due to  filled disk volumes.

Embrace automation, free up time for innovation!

Public Clouds such as AWS provide tremendous flexibility and a rich set of Platform features for users.   However, as one starts to scale up usage on AWS, day-day Cloud operations (Ops) could become a challenge. 

                     Here are some reasons why Cloud Ops automation on AWS is worth the investment:

a) The Sheer number of Infrastructure attributes on AWS makes Ops challenging.  Running Cloud Ops while staying on top of existing capabilities  (contending with AWS new feature releases) is easier said than achieved. Moreover, AWS is not just about Infrastructure. Users have to deal with Applications, Infrastructure and the resulting workloads.

b) AWS can be a complex ecosystem to tame and not everyone is skilled in Cloud Operational automation.    Effective Ops Automation (where possible) needs an approach that melds skill sets with clear objectives.

c) Developers and DevOps would love to free up their time for innovation (new features, applications, Business impact) instead of trying to manually solve Operations issues.  Example:  try multiple batch jobs manually on AWS at scheduled intervals.

d) Manual DevOps approaches directly impact Time to Business Readiness. Let’s say you had to test out a critical upgrade for your e-commerce site across 50+ Instances and verify if your Operations is ready to support this rollout. An automated Ops approach to test and validate across reduce your time to get go live and increase overall confidence

e) Limited ability to track the Cost impact of Operational activities at scale. Example:  Ability to schedule start and stop 50+ EC2 instances for an hour everyday and track the cost usage for that hour.  

Fortunately, help is available.  You could either invest in creating custom scripts to automate some tasks or use Cloud Automation products such as Botmetric to automate, track and resolve simple tasks. Here are five examples that you could consider automating: 

Taking Automated Volume Snapshots for EBS and RDS:  In the event of Disaster Recovery, it would be important to have EBS (Enterprise Block Storage and RDS (Relational Database Systems) snapshots in place to aid recovery and restoration.  You could either write a script to automate these tasks or use Botmetric, which provides a simple one click “ Cloud Fix” capability that can automate taking snapshots based on AWS volume tags or instance tags.

Enabling Termination Protection for AWS EC2 instances:  Termination protection on AWS prevents accidental termination of EC2 instances. Imagine if you ran a Security audit and realized you have to activate Termination protection on 100+ EC2 instances!    To automate this task, you could either write a script that detects and activates the protection or use Botmetric to discover, Visualize and automatically enable Termination Protection via a single click.

Releasing Unused Elastic IP address:  Elastic IP addresses on AWS cost you money if unused.  Botmetric can detect unused IP addresses across your regions and provide a one-click button to release unused IP addresses.

Run Batch Jobs at scheduled intervals:  If you needed to start and stop EC2 instances based on scheduled time of the day, Botmetric provides a simple way to schedule such jobs as alert users once the task is completed. Also, you can track the cost of the Workloads.

Removing Unused ELB (Elastic Load Balancers): ELB are not cheap and often, customers don’t realize this until they have accrued a significant cost. You can use Botmetric to automatically detect and remove unused ELB.

All of the examples above could be achieved using manual approaches however taken in aggregate or done  individually , these mundane yet repetitive tasks take a significant toll on your DevOps teams. 

We strongly encourage users to automate where possible either by writing custom scripts or by using Cloud Automation platforms such as Botmetric.   As a reference, our own experience tells us automating   Start and Stop of EC2 instances reduce Operational fatigue by an order of 5X  and gives back significant hours to Developers  and DevOps teams which they can invest in creating better products rather than doing mundane  EC2 start/stops.  

Start Automating Now! Signup for Botmetric.

There are plenty of reasons to move to the cloud, but one of the key benefits from public cloud providers like AWS Cloud is pay-as-you-go pricing model with elastic scalability to grow or shrink usage based on your business needs and growth.

(Image source: internationalnetworksolutions.net)

However, as you start using AWS Cloud, you will notice that the cost spikes and spend trends going upwards. So it's important to have right tools like Botmetric for your cost monitoring and analytics which can keep of changes and control your costs.

This blog post outlines the top 5 cost saving tips for AWS cloud infrastructure. Here are our recommendations listed below and we would love to hear your feedback:                                         

Use New Instance Families and Types - If you are using AWS for few years then chances are that you are running old generation instance families like m1, c1, t1 etc. If so you should move to m3, t2 and c3 family instances as it provides better CPU/Memory combination at lower price compared to old generation instances. We have seen upto 10 to 20% cost savings for customers who adopt to new generation instance families and get better CPU/memory combination for the given price point.                                                                   

Shutdown DEV/QA Workloads on Weekends - One of the smart strategies for cost saving is to shutdown the DEV/QA workloads during weekends. This would require your services to be bootstrapped to start/stop automatically when you start/stop your EC2 servers. Cloud automation tools like Botmetric support automaton jobs feature to shutdown and start EC2 servers based on instance tags on Friday night and bring them online on Monday early morning. It only requires few hours or days of effort but it can help you save anywhere 15% to 20% of the EC2 costs for DEV/QA instances.                                                                                         

Caching and Compression for CloudFront Content - From the analysis of Botmetric cost analytics for hundreds of AWS accounts, we have noticed a pattern of modest to huge CloudFront spends. Most of the times, customers would have forgotten to enable caching for CloudFront resources either at origin servers like Web or S3. You should enable compression for content like CSS, Javascript, Text Files, JSON, HTML etc to ensure  cost savings on bandwidth. Also, frequently deploy production changes and often forget to enable caching & compression for static resources, dynamic content like text/HTML/JSON etc. This recommends to have post-deploy job as a part of your release automation to ensure client side caching, serverside compression etc are enabled for your application and resources.                                                                                                    

Remove Unused Resources - In any typical business, different teams like product, QA, IT, automation and developers are using AWS cloud for developing, testing, deploying and managing applications. We have seen many unsued resources like volumes, elastic IPs, old snapshots, unused data in S3, idle load balancers, heavily under utilised machines etc. Tools like AWS Trusted Advisor and Botmetric provide insights into unused resources so that you can clean them up for saving cost. Botmetric also provides automation jobs to ensure you keep track of cost optimisations due to unused resources and one click jobs to remove them.               

Use AWS Reserved Instances - One of the best ways to save cost is to reserve the instances (EC2, RDS, Redshift etc) to reduce costs from 20% to 60%. We have outlined the AWS RI plans with recent changes in earlier blog post and a good RI planning can help you save significant cost. There are many cost analytics tools like Cloudablity, Cloudyn and Botmeric to help you plan your RI purchases. We strongly recommend businesses to evaluate their RI utilisation and purchases on a monthly basis if you have growing usage of AWS inside your company.

To sum it all up, you need to have a good review process and timely alerting internally along with smart tools ranging from AWS Trusted Advisor to Botmetric etc for effective cost managing and savings.

You can do this and lot more with Botmetric, get started with free trial and our plans start at $99 per month.

Cloud security is one of the business concern areas for enterprises adopting public clouds like AWS. In this blog post, we will outline the top 5 security tips that are 'must implement' for protecting your AWS account and securing Cloud infrastructure.

Botmetric provides Cloud Insights feature to keep track of AWS security best practices on continuous basis and provide you with guidelines for resolving identified security risks.

The below recommendations are must have for your AWS account:

Disable Root Account Keys - Every AWS account will have root account along with access key and you must disable/delete root account access keys. We also recommend a lock down of root account and use IAM admin approach as mentioned in next step.                                                                                                         

MFA for IAM Admin Accounts and Root Account - We strongly recommend to enable multi factor authentication for all IAM admin accounts and Root Account. You should have atleast two IAM Admin users per AWS account to ensure there is no single point of failure. We also recommend you to lock down root account by enabling MFA and removing all root account access keys.  You can create specific IAM role access keys with required policy permissions.                                                                                                                                                                               

Enable MFA Delete for S3 Buckets with Business Data - AWS S3 is a great service for storing business data including important database/filesystem backups etc. We recommend you to enable MFA delete protection on S3 buckets to ensure critical business data and backups are safe guarded from accidental removal or due to access key compromises etc.                                                       

Bastion Host for SSH/RDP Access - Instead of allowing direct SSH/RDP access to all your cloud servers, it's recommended to use bastion host approach for allowing SSH/RDP access. You can also enable multi factor authentication for bastion host SSH access. Only trusted IPs/networks should be allowed to SSH/RDP to the bastion host and using AWS Security Groups, you can further allow SSH/RDP access from bastion host to the required EC2 servers.                                                                                                       

Use IAM Roles Over Access Keys - You should use IAM roles to provide the temporary access keys for programs or applications or scripts running in EC2 instances instead of providing permanent access keys or hard code them via configuration files. With IAM Roles, you can restrict the access to specific machines and provide temporary access keys to avoid key compromise issues. Also, for any third party applications, provide Cross Account IAM Role Access (3rd-party) instead of giving permanent access keys.

We recommend you to read AWS security best practices and follow their security blog for periodic updates on AWS cloud infrastructure security.

AWS has launched a simplified Reserved Instances plan by removing the earlier complexity, you would have received an email from them earlier this week.

Please find below a quick overview of the proposed changes:

1. No Upfront Fees - You can get upto 30% overall savings compared on-demand price by going with this plan. However, you would commit to pay for RI rate over 1 year period. This plan is available only for 1 year RI period.

2. Partial Upfront Fees - You can pay part of the fees upfront and get upto 60% savings over on-demand plan for 3 year RI term. This is our recommended plan as it best suits for most businesses and workloads on AWS.

3. Full Upfront Fees - You can pay full fees upfront and get upto 63% discount for 3 year RI term. The upfront fees are almost 2X for this plan compared to partial upfront plan. For certain workloads like databases, search servers on EC2 etc can be reserved under this option.

Botmetric recommends looking at option 2 as you can save 50% in terms of upfront fees and cost benefit is almost as good as option 3. Also we strongly recommend going with 3 year RI plan instead of choosing one year RI as it's best suited for most scenarios.

We are happy to announce that our client agent is certified for Windows Server 2008 R2, 2012 R2 platforms.

Botmetric agent supports automation of common tasks like enabling SWAP, cleaning up log files for archival, enabling NTP time synchronisation and detecting unallocated disks etc.

We are improving our agent intelligence to help you solve common operational tasks through automation so you can focus on your application or product instead of doing system administration tasks.

We are happy to share that the first version of CloudFix is launched to provide an auto healing for issues like unused AWS resources(EIPs, Volumes, ELBs etc) and perform required backups for your cloud storage without the complexity of custom scripts.

CloudFix requires specific permissions like create snapshot, delete snapshot, delete unused EIP/ELB/EBS for providing quick resolution and help you save cost. It's safe to use CloudFix and you can enable Multi Factor Authentication for your Botmetric account to get an additional security protection.

We have improved Budget Alerts feature in our Cost Analytics to help you receive email notifications based on daily spend, weekly spend or projected cost for the month to have a handle on your AWS cloud spend projections. With Botmetric, keep your costs under control.

Try us out and share your feedback, we are working towards simplifying the life of Ops engineers. 

ENJOY A 30-DAY FREE TRIAL AND NO CREDIT CARD REQUIRED!

It's been an action packed week for Botmetric team with great traction for our product launch at re:Invent.

We had hundreds of visitors to our booth and enjoyed all those conversations during sessions, lunch, breakouts and re:play party.

Top 5 AWSome announcements at re:Invent:

1. Lambda - We loved this feature and will be incorporating it to build a smart tasks processing layer instead of managing worker instances in Botmetric platform. We welcome Lambda to AWS.

2. Aurora - Scaling and managing large databases for performance is a difficult task in Cloud as well and it just got dead simple with Aurora.

3. Faster and Larger EBS - We hate to build and manage RAID volumes or LVM snapshots so the new 16 TB EBS volumes will be loved by Cloud Operations team.

4. EC2 Container Service - We do so many deployments every day at Botmetric that we needed a smarter and easier way of doing this in AWS. It's time to explore ECS.

5. Code***** - It's great to see CodePipeline, CodeDeploy and CodeCommit from AWS. Our developers will love the new way of doing things. We can get better and smart at DevOps!

We enjoyed meeting so many interesting people and all those crazy parties in Vegas during re:Invent. Here are some pics :)

Thank you AWS and Vegas, will be back in 2015!

                                                                                   One of the biggest technology trends today is migration of On-premise infrastructure to Cloud. Companies enjoy enormous benefits from Cloud in terms of cost, flexible, scalable and elastic features. But, that’s not the end of the story.

Wondering what’s next?

Post migration, you will need a team who can manage and monitor your cloud infrastructure efficiently. As your business grows, the complexity of your cloud operations increases so businesses need an efficient way of managing this complexity.

Here’s that smart bot, meet Botmetric, your virtual cloud engineer who can help manage your cloud operations better.

Botmetric is an integrated operational agility platform for public cloud services. The virtual cloud engineer delivers proactive operational insights, executes repetitive IT tasks and offers smart recommendations for optimizing cloud spends. It blends operational intelligence with automation to help you make more informed decisions.

So, cloud architects and engineers, IT managers and business heads, make your life easier by using Botmetric. 

If you are going to be at the AWS re:Invent, you can meet the team there and know all about Botmetric with a free demo (Or) write to us at [email protected] for trial account.