#technicalnarrative

Before you can actually write a technical narrative, you need to know what to write about. And if the technical data you’re working from isn’t great, your technical narrative will also suffer (and perhaps attract an HMRC enquiry). Garbage in, garbage out.

So, to increase the quality of your technical data, and minimise the garbage-factor, you need to know how to construct your narrative in the most HMRC-friendly way to maximise your chances of reclaiming R&D Tax Credits, and this post will explain exactly how to do just that.

Questions, questions, questions: what to answer and how to answer them.

Whether doing the technical data collection in person or via email, it’s good to generally focus on the following key questions for each project. A sample data collection form is included further down.

Name of the project

It’s obviously useful to have a name to refer to the project by, whether it is “Microsoft Outlook PGP Support” or some sort of code name.

When did the work start and finish?

As covered earlier, projects need to be delivering new stuff every financial year. Asking this question:

makes it possible to check that the project qualifies in terms of when the work was performed;

can be used to justify why the project was included in case HMRC challenges this;

helps the writer determine how much time to spend on the project (if a project ended 2 months into the period, and was covered in the previous claim, it probably doesn’t need that much new detail).

Rough (month) dates are of course ok for this question.

What is your project? What are its capabilities, summarised?

The point in this section is to help HMRC understand what the project was about, in brief, to put the rest of the technical data in context. It should not be very long (a couple of paragraphs are enough), but it should be clear. It may include a brief mention of the competitive landscape that made the project necessary.

As an engineer, can you describe the design, execution or production challenges that you faced?

What HMRC will be looking for are actual design challenges instead of process problems. They are looking to hear about algorithms not meetings discussing algorithms. They don’t want to get into the details of the solution just yet, just outlining what the problem was in detail.

What platforms and tools did you use to make it?

This section is also useful for the writer to put things in context, and can make it easier for the HMRC to dig deeper into your project if necessary. In this section you should be name-dropping languages, frameworks and components.

What was created/implemented to overcome the technical challenges you faced? Are there any hacks you're proud of?

This is where you really need to go into full detail about things that were tried to overcome the problem. You should also be sure to include details of attempts that failed, as evidence that the first few attempts to solve a problem failed for technical reasons are highly indicative of technical uncertainty, and HMRC specifically looks for these.

If you get any ideas about feats of technical brilliance (or even just clever hacks) that don’t really relate to the specific challenge you were covering, include them anyway. It’s best to err on the side of too much than too little.

What is the market position of your product/project?

More specifically can you please explain how it functionally differs from other competitors in the same class? What improvements and enhancements have you made to improve your competitiveness? This is helpful to the writer because it can help to present a case for why the company is innovative, based on comparison with its competitors.

Final tips about the technical data collection

If the tech data collection is done well, it enables the efficient use of external writers like the ones provided by GrantTree. If you just can’t find the time to fill in this data, we can come in and have a meeting to do it in person, and this in-person model (which we often call a “123 meeting” and at which we usually collect both technical and financial data) is highly effective for larger organisations where it can be difficult to prioritise the tech data collection work otherwise.

One advantage of separating tech data collection and writing of the technical narrative is that it enables the technical person providing the data to be very efficient. They just need to be in the frame of mind and spend 30-60 minutes bashing out everything they can think of about what went wrong in the project from a technical standpoint. A competent technical writer can then take this data and turn it into a clear technical narrative that conveys the advances in a way that HMRC inspectors who are not specialists in the field can understand.

This compares favourably to the option where the same person who has the tech data writes the technical narrative - because often, their time is much too valuable to spend it learning how to write technical narratives, and so it is more efficient to keep their time involvement to a minimum.

Sample Technical Data form:

Name of the project: Woobius - API serialisation layer

When did the work start and finish? January 2007 - October 2008

What is your project? What are its capabilities, summarised?

Woobius is split into a Flash/Flex application at the front-end (which deals with interactions with the user, display of information, etc) and a Ruby on Rails application at the backend (which implements a lot of the business logic and stores data). In order to communicate, those two layers have to exchange a lot of data whilst the application is running, about the state, metadata and contents of the projects, folders and drawings which the user is interacting with, and about updates made by other users to those same items.

As an engineer, can you describe the design, execution or production challenges that you faced?

There were several kinds of technical challenges that impacted us here.

The first was around the communications protocol being used to transfer the information. At first we used the industry standard for Flex application communications protocols, RubyAMF. Then our own custom-built encoder. Then we switched to using text encoded strings wrapped in AMF messages. Finally we moved to building what was becoming, by then, the industry standard, a RESTful JSON api. Each of those options had its own tradeoffs and was a result of the things we learned through trial and error in the previous steps. The primary concern, and primary difficulty, was performance, and each successive version improved the performance substantially

The second was around the live update of the information. In a multi-user system, it is important to show users only the latest version, and to reflect changes made by other users very rapidly, and to prevent one user overwriting the changes of another. This is particularly important in a system which includes as a selling point that it will reduce such communications errors and provide an accurate, ordered audit trail that can protect construction project participants from litigation.

The third challenge was around the size of the information and how to partition it. A single construction project can be very substantial, as it will typically involve 20 or more companies collaborating together, and each can have a whole substructure of folders (say, on average, 20-50 subfolders), each containing bunches of files. In addition, each will have any number of issues in the “vault”, which typically can number the 50+ range, each containing several dozen, or, in some cases, hundreds of files. In short, a single project can be comprised of upwards of 20 thousand objects. Constantly reloading this entire tree of data is not practical even on very good connections.

What platforms and tools did you use to make it?

Our full development stack was involved in this project, all the way from loading records from MySQL, to handling them in Ruby & Ruby on Rails, serialising them with RubyAMF and other technologies, caching with Redis and then simple text files, serving files lightning-fast with Nginx, receiving data in Flash/Flex and decoding it with ActionScript, and even integration with Javascript/jQuery to pipe data around various parts of the application.

What was created/implemented to overcome the technical challenges you faced? Are there any hacks you’re proud of?

Where to start!

Let’s begin with the first one, the communications protocol. The industry standard for Flash applications is a protocol called AMF, which stands for ActionMessageFormat. It is widely supported by Adobe technologies, both server-side and client-side, and it is the recommended way to communicate with Flash applications. Since Flex is built on Flash, this naturally meant our Woobius Flex client should use AMF to communicate. The only available library to do this in Ruby at the time was RubyAMF.

The problems began when we were trying to fix performance problems in sending the whole project (before we partitioned it in different chunks) and we figured out that even a relatively small project with a handful of companies, folders and files was taking about 2 seconds to render - that is, the main delay in sending the project tree to the client was the rendering in RubyAMF. After a close examination of the open-source code in RubyAMF we determined that its approach to rendering was horrendously inefficient. First, we tried making changes to RubyAMF itself, and did get some performance improvements out of that, but eventually we realised that RubyAMF’s encoder was just very slow because of how it was designed, and so we began the work of re-building RubyAMF’s encoder from scratch. This one was fully tested (unlike RubyAMF’s), and built with performance in mind. We took a lot of inspiration from the Python equivalent project (called PyAMF), and from chatting to its creators. PyAMF was built for speed and vastly more efficient than RubyAMF, but written in Python. Using the test suite that the PyAMF team had built, we were able to build a solid, spec-compliant and extremely fast AMF encoder in Ruby (it was more than 10x faster than RubyAMF’s in most of our tests). This was a completely new encoder, much faster than anything available in the Ruby world.

However, even having followed this path, we found that the performance was still lacking. On a hunch, we tried rendering different things through the encoder to see what it struggled with, and found that encoding binary, fragmented Ruby objects made of lots of different fields was the key issue. Encoding a single string, no matter how long, was very quick. We converted one of the key requests to using a custom text format wrapped in an AMF message, and found that this was considerably faster than proper AMF objects, and could be easily decoded on the other end.

Next, we soon realised that having a custom format made things harder on both the encoding and decoding end - better to use something standard like JSON, which was starting to take hold as the de facto text encoding format for data (above XML or YAML). This had the benefit that there were standard (and fast) encoders and decoders available in both Flash/Flex and Ruby/Rails. So we then proceeded to convert the application to using JSON wrapped in AMF messages.

Next, as we began to hit against limitations of the Flex framework itself, and to consider a shift towards a more RESTful application architecture (and in fact to design new parts of the application using this model), we realised that the AMF wrapper was unnecessary, so we removed that and rebuilt the APIs once again as plain old Rails RESTful requests.

Altogether these steps resulted in about 20-30x improvement in performance compared to the initial model of using RubyAMF to send properly formatted AMF messages, and vastly enhanced the APIs to make them more future-proof and more easily interoperable with a variety of clients. This feature ended up being actively used when we built an iOS app for Woobius.

For the live update feature, we once again first considered using Adobe’s recommended technologies to handle the communications, but found several scalability issues in this step which needed to be executed many times per minute across every single user:

each request took too long as it needed to check the database to see if the object had been updated;

the requests also took too long to be encoded and decoded; we needed this to be snappy and very light on the server;

occasional, random server delays due to other big jobs running at the same time would cause delays in the response and wreak havoc on the synchronisation of data.

These issues were amplified by the fact that Woobius was, at the time, running on an operational environment that was very constrained in terms of memory available, so having lots of small requests frequently hitting the server caused all sorts of production issues.

To solve this, we made use of one of Nginx’s less used features, which is its ability to be insanely fast at serving small files directly, coupled with the ability to transparently expose Ruby on Rails public folders via Nginx. We then stored a timestamp of the latest update to the relevant object, within a plain text file within a publicly exposed folder. Each file was labelled by the folder (for the object type) and id of the object, e.g. woobius.com/flags/projects/123.txt would contain the timestamp for project id 123. This flag file was then very rapidly updated whenever a change was made to the project object or to any of its child objects. The Flex client could then limit itself to using polling to check whether the flag had changed, every 5 seconds or so, and update the interface immediately if the flag was different from what it thought it should be.

This of course had some knock-on effects, like requiring us to build an internal framework to track when objects changed and what the dependencies between those objects was, or, indeed, doing the same thing on the front-end to correctly decide what to update when a timestamp changed, or when several timestamps changed at the same time.

There were numerous other edge cases in this attempt at building a synchronisation layer, but in general this approach worked as a whole and scaled up well.

The partitioning problem was closely related to the construction of the dependency tracking layer. We found that the best way to split the project tree was into projects, “dropbox folders” (unrelated to the file sharing product with the same name!), and “vault folders”. Retrieving the root project object provides the folder structure, and each folder provides a fresh copy of itself and all the files it contains. This way each request would only return, at most, for a very large conglomerate of objects (e.g. a project with many companies or a folder with many files and comments), a few hundred objects, which while large, was manageable from a performance standpoint. Of course, making this work also required the dependency tracking framework on the frontend.

What is the market position of your product/project?

When we began to build Woobius, there were other project collaboration tools for the construction industry, but they were entirely focused on enabling contractors to track and manage large projects. There was nothing designed for speed and ease of use by architects, which was the market we decided to approach. Other tools that focused on contractors included 4projects, BIW and Asite.

Some comments about this sample answer

This sample answer is very long and very clearly articulated. Yours doesn’t have to be so long nor so clear. It is also already very neatly packaged, and almost directly usable as-is when putting together the technical narrative. This is likely because the person who wrote it has experience of writing many claims and so knows how to present data very clearly!

You don’t have to write such a long answer, nor such a detailed answer, but the key thing to take away is the fact that this is written by a developer, for a developer, and going very deep into practical, technical concepts. If the technical jargon isn’t there, it is very difficult for the writer to do their job.

Working out which projects you want to claim for is like building the skeleton of your claim; if you don’t decide on these to form the basis for your claim, your narrative will likely be meandering and tricky for the inspector at HMRC to understand - and therefore more likely to raise an inquiry.

To avoid this happening, the key with defining this section is to remember what your teachers intoned to you at school: plan, plan, plan. If you take the time to reflect on the projects you want to claim for in advance, then it will be much easier to objectively see what you want to outline and focus on, making your technical narrative much stronger than had you dived headfirst into it.

What defines a project? Could be a single product, major features or major challenges in the build.

Typically, a claim will cover more than one project. However, the word “project” can mean a lot of things depending on the context. A “project” might be a whole product, as delivered to the market. For example, a project could be Microsoft Word. Another one could be Microsoft Internet Explorer. Another one could be Microsoft Outlook.

Alternatively, it can be the case that all the R&D developments of a company fall under a single outside facing product. Then, “project” changes meaning to focus on major features or major challenges while developing that product. For example, Microsoft might have had a project going on to add built-in support for PGP to Outlook, and another to add support for easily adding GMail accounts, and yet another one to update part of the application to use some new Windows APIs, and yet another to extract that dependency on APIs to a common interface to make it easier to port Outlook to the Mac. These are all also “projects”.

Keep it simple: focus on about 1-3 projects in a claim.

So what definition of “project” is correct? It depends on your business, on what you’re claiming. The definition of “project” basically represents a granularity of detail. Generally, you want to set the detail level so that you end up with 3-5 projects, and cover 1-3 of them at most in a claim. It’s also reasonable to cover a single project, so long as that’s a good presentation of what actually happened.

Why not cover everything at the maximum level of detail? For two reasons:

You have better things to do than write a novel about your R&D. Like doing R&D, for example.

HMRC inspectors are also very time-constrained and even if you write that novel, they don’t have the time to read it.

The ideal level of detail is one that produces a technical narrative that lasts about 2-5 pages of A4. More than that, and the HMRC inspector will either start skimming, or start getting annoyed, or both. Less than that, and the claim may still pass, but it is likely to raise an eyebrow (in other words, it increases the risk of HMRC inquiry, which as we’ve established is highly undesirable). In our experience, 3-5 projects produces the desired length, and 3 is often sufficient. A single project can do as well, but then it needs to have a lot of meat to it.