#robot framework

In large part because people are going to do the bad thing (have mutable state dependencies between tests) and I'd like to incentivize them to at least make it explicit.

But also because sometimes you've got a test like "Test equality overload" and then you've got a test like "Test that equality overloads are consistent with less-than and greater-than overloads", and if the first one fails then the failure of the second will almost always be redundant noise, so you want to be able to somehow tell the test suite to not bother running or reporting the second test if the first test fails.

So recently a couple design questions came up:

Test A depends on Test B, but test B was either omitted from this run, possibly because the human operator knows that the dependency is met, or Test B doesn't exist - let's say our code can't tell the difference between those cases. Should that be treated as an error that fails test A, or a warning that lets test A run?

Test X depends on test Y, and test Y failed. Should test X be skipped, or also marked as failed?

Robot framework is an open source service that enables users to automate test cases using keyword driven development. It consists of pre-defined keywords and also allows the creation of new keywords. Hence, the combination of these two can let the testers automate anything!

Keyword-driven testing has gradually gained popularity in automation test development because its primary goal is to create functions that consist of a set of steps to complete action(s) in performing a test case regardless of the test framework used.

The Robot framework encompasses automation testing for web applications, mobile applications, and also desktop applications.

So if you just write:

.. code: robotframework

A Test Case Name

Some keywords ...

that won't highlight properly on PyPI. It has to be:

.. code: robotframework

*** Test Cases ***

A Test Case Name

Some keywords ...

And that's every code block. Doesn't matter if the code block above it already had it, and you're just breaking it up with some explanatory text before getting to the next keyword. You have to repeat the section header in each separate Robot Framework code block.

Which makes sense if you think about it, because Robot Framework language is inherently not context-free. Most lines depend on a line preceding them for context which changes how they are parsed. For example there is different parsing in the Settings section than in the Keywords section.

Of course, why doesn't ReStructuredText code coloring just follow Robot Framework's own example? Robot Framework parses .rst files (great for literate programming with tests, although Robot's syntax already lends itself profoundly well to human-readable tests), and it doesn't have any trouble with each code block not re-establishing the section, because it first catenates all the Robot Framework blocks and only then parses them all as one. This is why it doesn't normally occur to me to add the section headers in each block - I run my README.rst with `robot` and if the test results are as-expected I think I've done all I need to do with the Robot Framework code blocks in the README.

In a previous post I showed some examples of how to connect to Sauce Labs in Robot Framework using both the SeleniumLibrary and AppiumLibrary. I also alluded to showing how to write custom libraries with Robot, which I will show in this post.

In my humble opinion, one of the killer features of Robot Framework in Python is the ability to write custom libraries using pure Python and the Robot API. This can be done in addition to using pre-written keyword libraries or instead of using such libraries. Writing custom keywords in Python that can be used in .robot files is also a neat separation of concerns for test writing and test maintenance. A team's Pythonistas can handle test logic and page object layers in a conventional way while non-Pythonistas (BA-istas or PO-istas perhaps?) can write test cases using a Markdown-like format.

Here is a a scenario where using a custom library instead of SeleniumLibrary may arise. As well, I'm going to use features in the Robot Framework API as of version 3.2.

How To Do It

Suppose a web app needs to be tested against a latest version of Safari - 12.0 or later. These later versions all communicate with the Selenium WebDriver using the W3C wire protocol, which drops desired capabilities and changes the capabilities/options syntax. There are also changes to how cloud services such as Sauce Labs receive capabilities with this new protocol. This scenario is (currently) a bit tricky to handle with SeleniumLibrary, but relatively easy to handle with the Python WebDriver bindings.

There are two parts to this test scenario, which are

the Robot test case, and

the Python custom library.

Let's examine the Robot test case first, which looks like this

*** Settings *** Documentation A test suite with a single test for valid login. ... ... This case was created to validate the app against Safari. Library SafariCase.py *** Test Cases *** Verification Workflow Open Safari Login As User Verify Page [Teardown] Close Safari

Let's say this file is called verify_safari_case.robot. So far this looks like a conventional .robot file test case. The only interesting part is that the library that is imported is a Python file. Let's take a look at the Python file SafariCase.py:

from robot.api.deco import keyword, library from selenium import webdriver import os BASE_URL = 'https://www.saucedemo.com' @library class SafariCase(object): def __init__(self): self.browser = None @keyword def open_safari(self, url=BASE_URL): caps = { 'browserName': 'safari', 'browserVersion': '13.1', 'platformName': 'macOS 10.15', 'sauce:options': { 'seleniumVersion': '3.14.0' } } username = os.environ['SAUCE_USERNAME'] access_key = os.environ['SAUCE_ACCESS_KEY'] sauce_endpoint = 'ondemand.saucelabs.com/wd/hub/' remote_url = 'https://{}:{}@{}'.format(username, access_key, sauce_endpoint) self.browser = webdriver.Remote(command_executor=remote_url desired_capabilities=caps) self.browser.get(BASE_URL) @keyword def login_as_user(self): self.browser.find_element_by_id("user-name").send_keys("standard_user") self.browser.find_element_by_id("password").send_keys("secret_sauce") self.browser.find_element_by_id("login-button").click() @keyword def verify_page(self): assert "inventory" in self.browser.current_url @keyword def close_safari(self): self.browser.close()

Definitely looking like a conventional Python file with a single class SafariCase defined. To execute this test case, we can use the Robot runner as follows:

robot verify_safari_case.robot

if we require that verify_safari_case.robot and SafariCase.py are located in the same directory.

All good at this point. Writing the test logic for keywords can be done using Python while test cases themselves can be written in Robot. Time to celebrate!

Now let's unpack SafariCase.py a bit further.

First we can note that we're making use of Robot's API for writing keywords and libraries via

from robot.api.deco import keyword, library

We only need to import the keyword and library decorators for our case but there are other tools in the robot.api module for other use cases. These decorators allow a standard class with methods to become a Robot library with keywords. Each method decorated with @keyword creates a Robot keyword that can be used. Robot in Python is smart enough to handle casing changes, so a snake_cased_keyword can be interpreted as a Sentence Cased Keyword without much trouble. This is helpful for keeping Python code looking like Python while Robot file look like, well, Robot.

The @library and @keyword decorators work mostly as one would expect but both take additional optional arguments for things like scoping. Also note that @keyword functions do not need to be in a class but we do so here because we need to maintain an internal state for the browser field. Object-oriented programming is not a requirement for writing custom libraries but tends to be helpful.

One last important point is that the name of the library class SafariCase matches the name of the Python file SafariCase.py. This is a convention and simplification in the Robot library API. Please follow this convention.

Warning: Mind the Gap

There is one important pitfall I will point out to writing custom Python libraries for use with Selenium- and Appium-based tests, something that astute readers may have noticed already. What happens if the SafariCase library is used in conjunction with test cases that also make use of SeleniumLibrary keywords?

One problem that comes to mind is collision of keywords. What if SafariCase implements a keyword called Open Browser, which is also in SeleniumLibrary? How would a test know which one to execute? More importantly, how would a test developer know?

A deeper problem is encapsulated WebDriver state. Remember that the SafariCase library uses an object-oriented approach by creating a class with a browser field. Would SeleniumLibrary have access to this field? If so, how? Managing WebDriver instances may get pernicious when multiple libraries are creating and handling their own instance.

These are both problems that can be solved by extending the SeleniumLibrary instead of writing custom libraries from scratch. This allows for custom functionality that can be added to SeleniumLibrary for a given project. I may or may not dig into this topic in a future post, depending on [gestures wildly] all of this stuff that happening.

I'm a major Python fan. One test tool that's gaining (has gained?) popularity in the Python world is Robot Framework. Robot Framework - which I will call Robot from now on - is a keyword-based test framework with a wide range of keyword libraries and a clean keyword domain specific language (DSL) for writing test cases that is quite similar to Markdown. One of the reasons teams choose Robot is that it allows developers to write Python code when needed but also provides neat and tidy approaches to working with both Selenium Webdriver and Appium in projects.

One question that comes up from time to time when using Robot with Sauce Labs is

How do I connect my existing tests to run on Sauce Labs?

In this post I'll cover two examples of how to connect Robot tests to Sauce Labs. First, I'll show how to connect Selenium-based web tests using the SeleniumLibrary library. Second, how to connect Appium-based native app tests using the AppiumLibrary library. In a later post, I'll show how you can use pure Python to create custom libraries, or at least the basics of how to do so.

For the rest of this post, I'm going to assume basic familiarity with Robot constructs such as writing test cases and working with libraries.

Connecting Selenium-Based Web Tests to Sauce

The most common approach to writing Selenium-based web tests in Robot is to use the SeleniumLibrary library. This library wraps up all needed Python dependencies such as the Webdriver and provides an extensive list of keywords to use for browser actions.

The main keyword I want to point out is Open Browser. This keyword starts a new browser session, either local or remote. It has a set of named arguments to pass in, and if no arguments are passed in will default to trying to start a local Firefox browser instance.

To connect to Sauce, the following arguments are required:

browser

remote_url

capabilities

which is usual for Selenium Webdriver sessions that connect to Sauce Labs. Here's an example of a test using the SeleniumLibrary keywords to start a Chrome session on Sauce Labs. Let's call is selenium_example.robot.

*** Settings *** Library SeleniumLibrary *** Variables *** @{_tmp} ... browserName: Chrome, ... platform: Windows 10, ... version: latest, ... username: %{SAUCE_USERNAME}, ... accessKey: %{SAUCE_ACCESS_KEY}, ... name: ${SUITE_NAME}, ... build: My-Selenium-Robot-Test ${browser} Chrome ${capabilities} ${EMPTY.join(${_tmp})} ${remote_url} https://ondemand.saucelabs.com/wd/hub *** Keywords *** Open Test App Open browser https://www.saucedemo.com browser=${browser} ... remote_url=${remote_url} ... desired_capabilities=${capabilities} *** Test Cases *** Verify Connection Open Test App Title Should Be Swag Labs [Teardown] Close Browser

This can be executed by running

robot selenium_example.robot

Let's unpack this.

First, the SeleniumLibrary is imported. This provides all keywords in that library for use in this test case.

Next we define some variables. Since the needed arguments are browser, remote_url, and capabilities, each of these are defined. Both the browser and remote_url arguments are strings and so can defined in a pretty straightforward way. The capabilities argument is defined as a list of values, and so a list is created using the Robot test case syntax, providing all needed information for Sauce Labs.

One subtle but important thing to point out when defining values is that Robot provides some special constructs for defining values.

To access Sauce username and access key credentials, the environment variable values are used (the % values).

As well, inline values can be computed/called using the scalar value construct (the $ value for name). Values within a scalar can be either Built-In values such as the SUITE_NAME above, or can be Python values either constant or computed.

Since we have the needed values to start a Sauce session, we can pass these values into the Open Browser keyword (note: there are two spaces between a Robot keyword and its argument(s)). I've decided to create a custom keyword Open Test App to wrap up starting a session in a neat and tidy way.

Finally, we have the test case. The test case is very basic, starting a session, checking a title then closing the session. Clearly this is a low value test but sometimes good developers write bad tests for demonstration purposes. This test makes use of the Title Should Be and Close Browser keywords from the SeleniumLibrary.

Voilà, we have a working test in Robot that connects to Sauce Labs.

Connecting Appium-Based Native Mobile App Tests to Sauce

A case that comes up even more often in my experience is using Robot to test mobile apps (web and native). This is likely because the AppiumLibrary makes getting started with testing native apps more intuitive than using Appium directly (but could also be because folks just love using Python!).

Much like the SeleniumLibrary, the AppiumLibrary library packages up Python dependencies for working with Appium and provides an extensive list of keywords for working with mobile apps. Let's take a look at a similar Robot test for connecting to Sauce labs to test a native mobile application and call it appium_example.robot.

*** Settings *** Library AppiumLibrary *** Variables *** ${KEY} %{TESTOBJECT_SAMPLE_ANDROID} *** Keywords *** Open Mobile App Open application http://us1.appium.testobject.com/wd/hub/ ... platformName=Android ... platformVersion=9 ... deviceOrientation=portrait ... browserName='' ... testobject_api_key=${KEY} ... privateDevicesOnly=true ... name=${TEST_NAME} *** Test Cases *** Valid Login with Standard User Open Mobile App Element should be visible accessibility_id=test-LOGIN [Teardown] Close All Applications

This can be executed by running

robot appium_example.robot

As you can see, this looks similar to the Selenium case. However, one important distinction is in the Open Application keyword from the AppiumLibrary library. This keyword opens the target application, but instead of having all named parameters in its definition, Open Application also takes a remote_url named argument as well as a Python **kwargs argument, also known as keyword arguments. This is a variable-length dictionary of key/values that AppiumLibrary will pass along to the AppiumDriver instance that is instantiated.The remote_url argument is fairly straightforward to understand.

Likely, these kwargs will be a set of capabilities specifying the mobile environment that the driver will be created within. There's no set requirements or restrictions on the kwargs that are passed into the Open Application keyword but consider passing in only "needed" values.

Just for fun, instead of defining needed capabilities in a variable, they are hard-coded in the keyword itself. This is true for some definition of "fun". As well, the legacy TestObject API key credential is defined using an environment variable.

The last thing to note is that both Element should be visible and Close all applications are keywords provided by AppiumLibrary.

Voilà, we have another working test in Robot that connects to Sauce Labs.