Monkop Blog

The alpha version will run your test on one real iOS device, capture test video, CPU, vMem and threads. You can now try this version for free on www.monkop.com or using your own account at console.monkop.com

Monkop iOS Beta release will include multi-device test runs, more performance metrics and a larger device farm.

Setting up Monkop for Jenkins

1.- Create a new freestyle job: 

2.- Create a command line build step.

Linux: “Execute shell” 

Windows: “Execute Windows batch command” 

setting the following commands inside:  

Windows: curl -sL monkop.com/cli | python - -k %monkopapikey% -a “path_to_build.apk” Linux/Mac OS X: curl -sL monkop.com/cli | python - -k $monkopapikey -a “path_to_build.apk”

3.-  Consider setting monkopapikey as an environment variable using EnvInject Plugin and injecting the variable as a password in your job:

Note: Environment variables can be accessed using the syntax $monkopapikey (Linux/Mac OS X) or %monkopapikey%  (Windows).

Prerequisites

Monkop CLI uses Python + CURL to use Monkop’s API in order to run tests, show results and also update build status (if needed). So the prerequisites are:

CURL: If you don’t have it, please install curl where Jenkins will run your build process.

Enable the use of the API getting your APIKey on your Monkop account under the ‘Add-ons’ option https://console.monkop.com/addOns.xhtml

Python 2.7 installed (installed by default on Mac OS X and most Linux distros) https://www.python.org/download/releases/2.7/

Adding Monkop tests to your Jenkins pipeline will let your team automatically get key information on each build regarding how your app behaves on real devices while evaluating mobile-side performance. Check out Monkop CLI for help with creating different tests or running automation scripts.

Moreover, the ability to know what is happening in the jungle of mobile hardware in terms of resource consumption and bottlenecks is crucial. Gathering this information early on in each development cycle helps us to avoid bad UX. That’s just one more benefit of adding Monkop CLI to your testing.

Setting Up Monkop for Travis

1. Enable Monkop’s API in your Add-ons section: https://console.monkop.com/addOns.xhtml 

2. Add Monkop APIKey as a Travis-CI Variable in Repository Settings named monkopapikey:

3. Update your travis file .travis.yml  We recommend using Monkop-cli in the after_success section, but each DevOp designs his or her own pipeline:

curl -sL monkop.com/cli | python - -k $monkopapikey -a “path_to_build.apk”

> Sample .travis.yml

language: android jdk: oraclejdk7 android:   components:      - build-tools-19.0.0 after_success: - curl -sL monkop.com/cli | python - -k $monkopapikey -a “path_to_build.apk” 

Note: According to travis-ci documentation, if Monkop-cli is invoked from a script step, any failure of the apk processing will make your travis build fail. On the other hand, if you set up Monkop-cli on after_success or after_script step, your Monkop tests will never affect your build result.

Pull request integration for GitHub

Monkop has a bot that contributes to your build steps and documentation with test results by automatically tagging / commenting on your pull requests inside your GitHub repository. All you need to do is give access to Monkop inside your GitHub account.

Using OAuth authentication method:

1. Enable github add-ons in the Monkop console and authorize to give Monkop these permissions:

2.  You will be redirected to GitHub in order to authorize application:

3. Then you will see your GitHub account info inside Monkop’s console:

Now all pull-requests in GitHub can be tagged by Monkop

To revoke Monkop privileges, click the revoke access button in GitHub add-ons section in the Monkop console. This will redirect you to GitHub settings so you can revoke access anytime:

Installing Monkop CLI

Monkop CLI is a Python (v2.7) script that handles all communication with Monkop’s API using your account (APIKey). Once you have it, just download the latest version here: monkop.com/cli

Command Line Interface Options

You can get help from the command-line tool by calling monkop-cli -h

> monkop-cli -h   Usage: monkop-cli.py -k apikey [-t time] [-w] [-a appfile] [-st type]                     [-sf scriptfile] [-c callbackurl] [-e extradata]                     [-s transaction] [-h] General:  -k apikey            Your API Key (https://console.monkop.com/apikey.xhtml)  -t time                Test duration in minutes (per device)  -w                       Wait for completion  -n                        Don't return error code on test errors (status_code                              will be always 0) Application:  -a appfile           Public URL or local path to application file Script:  -st type              Type of script file. Valid values: ESPRESSO  -sf scriptfile       Public URL or local path to script file Notification:  -c callbackurl    Public URL for notification purposes  -e extradata     Extra data to be defined by customer, this data will be                             returned in the notification callback Status:  -s transaction  Get transaction status Help:  -h, --help           Show this help message and exit

Run your first test

First you need your app ready, if you don’t have one, you still can test some 3rd-party app to see what happens: 

> python monkop-cli.py -k YOURAPIKEY -w -a “http://static.monkop.com/setup/apps_demo/Google_IO_2016_4.4.6.apk”

Download and run in a single line with cUrl

> curl -sL monkop.com/cli | python - -k YOURAPIKEY -w -a  “http://static.monkop.com/setup/apps_demo/Google_IO_2016_4.4.6.apk”

Prerequisites

Enable the use of the API Key on Add-ons: https://console.monkop.com/addOns.xhtml

Python 2.7 installed (already installed on most Linux distros and Mac OS X) https://www.python.org/download/releases/2.7/

If the build runs over Windows, you may need to install CURL, and to have the CURL binary on same path of monkop-cli.py or accessible in path environment variable: https://curl.haxx.se/download.html

Limitations

When building apps, developers must find out which devices will be needed for testing purposes, or at least figure out how to partial release the app (region / devices / OS). Depending on the target market, phones, tablets or Android version, to know which devices are needed is getting harder every year. Developers usually start testing their apps on some emulators, but this is only recommended for an early stage, why? simply because apps behave different: App incompatibilities, unsupported devices, and bad performance literally kills the app on the marketplace, contributing to the top reason on why majority of businesses don’t cross the “app poverty line” (less than u$s 500/month in revenues)

Today, it is important to have your own devices to test the first bytes, but after that, having devices at home is not a “sexy” option, even for for businesses. Let’s see:

Having 50 devices will reach only about 25% of the market

Charging / maintenance, hubs location.

Network stuffs (you need to feed devices with WiFi / Carrier)

Deploy system / servers (if you will need to run regression tests or share devices)

Devices management about each device location / OS versions, ids, etc. start to be a pain.

Devices to cover US market are very different than cover Asia.

Market changes fast! Soon, you will need to add more devices.

Let’s take a look at top insights in Open Signal report:

Number of brands seen in 2015 are near 6 times in 2012.

24.000+ different devices

Over 1000 new brands seen (not seen in 2012)

Android OS is still very fragmented on versions

source: http://opensignal.com/reports/2015/08/android-fragmentation

Companies like Amazon have recently launched what we (and many others) were expecting: a platform to rent devices on the cloud. But this is only the beginning, developers needs more tools, platforms and solutions to know where they should take a look to fix what is important, how can they fix it and which devices are crucial to choose depending on budgets. That’s why we are happy to be in this trip, fully committed to provide affordable and intelligent tools to help developers and testers to be successful on the Android market.

Overview

This document explains how to understand Monkop report after each execution (test-ride). This is how it looks like:

The report is a web page, so it can be accessible from any browser. It contains a left navigation panel and dashboard overview.

Navigation Panel: The left menu contains all sections and highlighted warnings and cautions. This panel enables users to navigate across:

-       Dashboard: Contains market stats and selected devices

-       Executions: The most important section, containing each device information like OS version, screen and hardware details, and also execution-related information such as: videos, screenshots, installation process, timing measures, resources and logs.

-       Information: General information about the application under test, such as: version, compatibility, permissions requested, activities and included libraries.

-       Correctness, Performance, Resource usage, security and power sections: each section containing analyzed behavior over all devices. Yellow / red indicators are values out of Monkop’s default thresholds, which are based on Google best practices and important market researches.

-       Errors: Contains crashes reports, and logs when app is not responding (ANR).

This section contains a market chart, execution details, device selection segment (by screen density) and test-run information.

The following chart shows the latest Android market stats based on data collected by Google.

Executions

This is the most important section, since it has each device-specific execution. It shows executions that user has manually selected, or those one that Monkop has automatically selected for him

Clicking on any device, users will find device specific information about execution:

Below this information users will find all details about device performance, videos, screenshots and logs.

Keep updated to learn about how to understand Monkop’s indicators. That yellow/red bubbles you find on each execution.

Emulators run on a PC, so they emulate devices such as mobile phones in a generic way over a totally different hardware, which is where the app market is. So, performance evaluation is useless.

Official Android emulators emulate a set of devices in a determinate operating system, but real brands have their own Android implementations and they behave differently.

Some important apps don't work on emulators due to various reasons, such as security, debug modes or architecture incompatibility.

Most emulators only provide support in terms of general software stack, without providing common services such as Google, which are required by many apps.

App users don't run apps on emulators, they run them on real devices, so you want to test on the same devices users interact with, in order to have accurate tests. Remember: “0% of your customers run your app in emulators”.

App Testing on Real Devices

Testing on real devices is much more likely to produce the kind of results you need to launch a successful app. However, there are some issues that need to be addressed with them as well –

here are three:

Depending on your market, you might have to test on dozens of real devices and buy new ones on a regular basis, in order to keep completely current. Buying many devices – including the latest ones as they are released – obviously could be costly.

Another issue – it takes time to use many different devices in terms of starting them up, opening an app and running it. Also, those devices have to be plugged in, charged up and stored properly, with their power cords nearby and easily accessible. None of these activities is difficult, but combined all together, they eat up time.

Of course, you could borrow devices for testing if you know people who have them, but that requires some coordination and time too. Additionally, if you break a device that is on loan, you would most likely have to pay for the repair or replacement cost.

The solution to these problems is to use cloud labs – using Monkop enables devs and testers to upload their app into a cloud environment, which connects to a variety of physical devices in a smart selection of different API levels, screen sizes/densities and brands. You can choose which devices you want to test on, or test on several devices simultaneously. We are now offering a free service, so you can try it before paying.

Why Effective App Testing is Critically Important

You don't want to test in ways that aren't productive, because the results could be misleading. If the results are not accurate, your app will not be ready for a public launch, because it will have too many issues or errors. A faulty app will be a failure with users, and all the money and effort you put into it will be wasted.

You would never want to launch an app to the public with any problems, because users would quickly figure out they couldn't use it and bad online reviews would most likely torpedo its success. Also, there are so many devices to test for today, with a range of operating systems, that an emulator might not be very helpful.

The market for apps has been predicted to reach $77 billion by 2017, with one billion phones sold in 2015. Anyone with an idea and some tech savvy can create an app, and because the stakes are very high, the market is already crowded. The way to stand out from the pack is to only launch apps of the highest quality in terms of functionality and user experience.

If you had hopes of launching multiple apps and basing an entire business on them, that opportunity might also be wasted. Your app is too important to be trusted with a testing method that is not very effective.

It should be noted that bad written reviews are not the full extent of disgruntled user expression. Some users that are unhappy or upset will tell their friends about their bad experiences. Others may share their opinions on social media.

If you want to avoid having negative views about your app expressed in all formats, one of the best things to do is to thoroughly test it using a testing service like Monkop, before you launch. It probably also be very beneficial to test it on the largest number of devices possible. Below are nine causes of bad reviews or even app deletion.

1. Freezing

A survey of 500 consumers found that 76% of the respondents said an app that freezes might cause them to write a bad review. (This percentage was even greater than the number who said crashing would cause them to do the same, which was 71%.) If an app froze for just 5 seconds, 18% said they would delete it from their device.

2. Crashing

We have all used software or apps that crashed too often, and may have even lost important work due to crashes. User frustration is highest when an app crashes often, so an app should not be publicly launched until it is stable.

3. Bugs

Bugs are also one of the main reasons users become upset with an app, and will vent their dissatisfaction in online reviews. A buggy app will not generate positive reviews or many downloads.

4. Draining the Device Battery

Battery life is one of top issues for mobile devices, so any app that burns up a lot of energy and drains the battery is going to irritate many users. In fact, a consumer survey found that heavy battery usage was one of the main reasons a user would delete an app.

5. Bad Performance

If an app has a very slow load time, users will probably respond negatively. (One survey found that slowness could also cause users to delete an app.) Similarly, if the app doesn’t do what it is supposed to, users will become upset and may write negative reviews. Testing performance throughout the development process is more effective, instead of only doing it just before launch.

6. Poor App Localization

Research has demonstrated that another cause of negative reviews is poor localization, “Over 1/3 of all users experience poor app localization.” Source: Consumer App Usage & Preferences, Park Associates and Google, November 2013 Google Confidential and Proprietary

7. Difficult to use

Ease of use is one of things users like most - if an app is difficult to use then it will probably have a low adoption rate, negative reviews and high abandonment rate.  Launching an app before it is ready is not likely to result in success. You want to anticipate problems and solve them before users ever interact with an app. Test it before launch and gather feedback  in order to refine it before going live.

8. Poor User Interface

If an app has counter-intuitive menus, for example, the user experience will be difficult and they may lose interest if too much effort is required to learn the interface. Additionally, about 6% of apps submitted to the Apple App store are rejected due to poor design.

9. High Memory Usage

Apps that require an unusual amount of memory can be slow and cause crashes. Users don’t like slow apps, and some will not tolerate them at all.

Large and Growing Market

There are more Android devices in use worldwide than iPhones.  According to a recent article in PC World, Android has 52% of the market, while iOS holds just 15% of the smartphone market. (These statistics include Symbian OS as smartphones.) Open source phones are responsible for there always being competition to bring down the price of Android phones. Some Android phones are already available for under $100 without contracts. That price point means that Android’s market share will only grow, giving your app a larger playing field and growth potential.

As far as device sales, some changes in the market are predicted, but not ones which will necessarily bring down sales. According  to mobile phone forecasts by the international data corporation (IDC) worldwide quarterly mobile phone tracker,  Android OS devices will continue to drive shipment volumes, while iOS will continue to drive revenues in the current period. By 2018, Android will control 80% of global smartphones shipped, and 61% of the revenues. As markets mature, many Chinese upstart players in the global market will drive prices lower. However, premium phones for both Android and iOS are expected to continue their popularity as well.

If rapid adoption is what you’re after, launching on Android will give you an advantageous position, especially in the emerging markets.  

Flexible

Android is very open from a cross platform perspective. The software development kit works with as many third party platforms and non apple devices as you want. Its very quick and convenient for a company to get a Windows or Linux app onto Android. The designers meant for it to have multiple platform collaboration, and therefore don’t require a waiting period or a submitted application.

Customizable UI Offers Productivity Benefits

The user interface of Android offers much more flexibility and customization. Google has now loaded Android with a ton of customizable widgets. These can be placed anywhere on Android's UI and provide services such as email, calendar, and social media feeds. This is important because users can tweak their smartphones UI to meet professional needs on Android.

Development process

Android offers a very convenient development process.The Android platform itself is open-source, so developers can dive into the platform code when trying to find out troubling issues. Also, many architectural best practices are graciously made available by Google, such as the 2012 Google I/O (many more have since been made available) Cameron Henneke, 36 years old and a 12 year veteran software developer and programmer chronicled an app experience comparison:

Henneke spent a week reading books, following tutorials and creating test apps that implemented all the core functionality. Henneke then began architecting the app and writing the first lines of code. Within one more week he could develop comfortably on Android without relying on another resource to implement each feature.

Following a similar process for iOS, he adopted the same process but took two weeks of experimentation before he could even read the initial app code base.  

Even more time consuming than this was developing a scalable architecture for the "fetched results controllers" that backed the tables of the tasks. It took another two of weeks of development before he became comfortable coding on iOS.

While these were benefits he cited from his own personal experienced, many also prefer iOS.

Rotation for example, was actually easier for Henneke on iOS. “iOS requires just a punch of effort”, while the platform completes the rest of the process. Supporting device rotation on Android takes considerably more work and is the source of many bugs.

Publication

Android apps are extremely convenient when it comes to publication. Simply sign into your app via the eclipse wizard, and you immediately have an APK file that runs on any device. You can also immediately upload a big fix if a line of code caused an error.

Monkop.com provides fast and immediate app testing for a wide range of Android devices. These features include general information with videos, screenshots, correctness, performance statistics, resource usage, battery drain, database performance, real executions results, general information, and much more. To read more check out our website!

Version compatibility (API level set-up)

Most common and hard-to-fix mistake. “Jack of all trades, master of none” - chances are, when you build an app that works equally well on all Android versions from Cupcake to Lollipop, it will suck on each of them. Supporting old versions like Froyo or Gingerbread is not easy - users want performance, and cool modern apps don’t really perform that well on old devices.

You normally only make this decision once in your app’s lifecycle (when defining your minSdkVersion and targetSdkVersion in the app manifest) and then just forget about it.  

Solution: Before setting it up, spend some time with your team defining your target market in terms of devices, geography, demographics and other details that are relevant for your business. Android stats dashboard can help figure out the world fragmentation and track how users are moving. Also, remember that targeting the U.S. is different than Europe in terms of API levels and devices. Consider to set minimum API level = 14 (supporting Android > 4.0 Honeycomb) as a default.

Huge images

Sleek images make your apps look sexy, but they also consume a lot of memory. A single image taken with a 10 MP camera needs 40MB of RAM to be displayed, and the device quickly runs out of memory.

Solution: Phone display resolution is usually lower than that of high resolution images, so your users can’t really experience all your fancy graphics in full swing. No need to eat up their device memory for something they can’t enjoy anyway. Here are the three easy steps to make sure your app looks great and runs easy on the memory:

●       Be precise with your image sizing, ●       Scale/crop your image to fit, ●       Show only what can be displayed.

Personal information protection

You don’t need to be a hardcore hacker to gain access to Android app files or to sniff SQL queries. It is easily done even without rooting a device or decompiling. Here are some tips & tricks to consider for keeping sensitive information safe:

●       Do not reinvent the wheel: social network integration is sometimes the best solution. ●       If you prefer something else, protect sign-up process using Android Account Manager and try to use HTTPS for your back-end requests. ●       Try to use tokens. Or, at the very least, encrypt usernames and passwords. ●       Encrypt everything that needs to be safe in your mobile database. ●       Don’t save unencrypted personal data on the SD card.

Following these recommendations won’t make your apps top grossing, but you’ll rest assured you are avoiding the most common mistakes that have stopped other developers on their way to glory. You’ll also help make the fragmented world of Android apps a little more friendly for your users. 

Step 1) Just go to the project (app) you are working on:

Step 2) Then click on “View test data” button:

Step 3) Click on “Edit Form Data”:

Step 4) Put your values on the form:

In these days, in which all soccer enthusiasts gather up to enjoy and celebrate its most important event, software could not remain aside from it with apps flooding the stores, ranging from wages, statistics, analytics, live streaming, sharing photos, comments, up to a seemingly endless list. Those apps are widely used around the globe, therefore, the risk for success or failure is enhanced.

FIFA’s official app is presently getting plenty of negative reviews from end-users (see images taken from Google Play). Furthermore, it is interesting to note how the app is being well taken care of regarding certain aspects bringing quality to the mix, whereas very little attention is being derived upon others. Some features standout from the crowd such as UX, which not only does it look appealing, but also provides a good deal of information in terms of functionality. However, many users complain about unexpected behaviours, sluggishness in response times (performance) and unexpected shutdowns (robustness).

We seize this opportunity to share the results from the automatic analysis that Monkop conducted. It shows that those problems could have been detected and resolved in time and, for a very low cost. Monkop found, among other issues, that FIFA’s application was consuming a large amount of memory, it would be slow to respond, and at times, it would crash (depending on the device).

To begin with, there were occasions in which it would take up as much as 118MB of usage, even also reaching 125MB in heap size. In the graph you can see the most common heap size limits (16MB, 32MB, 64MB, 128MB) for which the application would crash.

Additionally, Monkop found that the application downloaded 7MB of data in a very short period of time, which in a 3G connection might turn into quite a number if sustained over time.

From Monkop’s report, FIFA’s app sluggishness is mainly related to two factors, on one side, as certain actions are performed in the application some frames are not properly rendered. On top of that, throughout some operations there was a high CPU usage, which, might make the experience to vary across end-users depending on their device’s hardware. Apart from that, there were some activities with a high degree of complexity in its layout which had a direct impact in both CPU usage and rendering time.

Furthermore, several FIFA’s World Cup end-users reported downtimes, which might be related to the devices itself having a small amount of memory assigned for each application, as opposed to the amount the application takes up, after which, it closes automatically after having reached an outofmemory.

We live in a world far more connected than ever before, smart devices have taken over and continue to ramp up, end-users have become demanding when it comes to an app, not only over what they want, but also and most importantly, over what they expect from it. Quality comes into the picture, having a direct impact on the success of your application, affecting from the amount of downloads and installations it gets, to the engagement and the retention rate it achieves from end-users, as well as its ratings and reviews from them, as seen below.

Try as we might, it is impossible to fully test a program. There are several reasons for upholding such statement, to name a few:

Impossibility of manually testing all the inputs in an application;

Not enough time to test all the combinations of inputs, and even if we did, we would have to evaluate the cost of engaging into such an activity;

Smart devices are fairly accessible to nearly anyone, therefore, making it far more complicated to tell what OS version and hardware (to name a few) should be in scope, and what not.

Lack of tooling and expertise to use them.

Having help to assist you throughout your testing activities becomes paramount for your success, Monkop gives you peace of mind. All it needs is your apk, so that it can analyze your app to provide you with guidance to improve its performance in terms of resource consumption, time response and rendering rates, giving you pieces of advice in terms of functionality based upon Android best practices, and on top of that, it is cross-device.

To sum up, be aware, Monkop is more than either a beautiful graph or a way to see the different data about the behaviour of your system under test. Monkop does much more than that: he analyzes the data, looks for problems, and also tells you how to improve those findings.