#openpatentdata

Author: Tim Jagodzinski

Six months in hindsight

Open Patent Data is the effort to introduce patents to the mainstream. To move dull boring technical descriptions buried in boring and often confusing official websites into a 21st century web application. When I was introduced to the project I was all excited (and still am) because I believe strongly that if patents would be read by other people than professionals, we wouldn't have all those silly general all-purpose patents polluting our world.

We went through half a year of excitement, blazing progress and crushing setbacks. Through great teamwork and terrible miscoordination. Basically I had the best six months in my career and I want to share this experience, so that you can draw whatever wisdom you like from it.

Kickstarting - like a kick in the groin

Our objective for this project was to build a state-of-the-art web application. It was supposed to be beautiful, efficient and accessible. We wanted to blow all the other "professional" apps out of the water and show the world how a 21st century patent information app is supposed to look like. We wanted to embrace agile development and use the shortest possible iterations to give us the freedom to just try out things or throw away stuff that wouldn't work.

There was only one little problem. None of our team had any notable experience with building web apps.

So if you don't have the experience you get people how do and let the teach you. That's what we did. And we got the talented people of Edenspiekermann(ESPI). We spent the first three weeks of the project with these guys had a great time, produced a complete project bootstrap and learned a lot about our agile workflow which helped us a lot in the long run. With so much good things, there must be bad things to balance out, ain't it? There was :)

We ended up with a great base for our project. We had a working backbone.js / Django tag team. And we didn't know Jack about how the freaking thing worked, let alone all of the little things of backbone.js and CoffeeScript that would blow into our faces every now and then in the upcoming months. Code documentation was sparse at best and virtually incomprehensible for us. It took us 2-3 months to get comfortable with the front end code base. Even today there are still dark corners which we don't fully understand.

Who's to blame? Can't really say. I think the problem was twofold (as so often). We didn't clearly enough convey our demand. Getting taught. And the ESPI guys didn't seem to have much experience in teaching. They were very focused on getting stuff done. An attitude that we adopted for the project (and learned from these guys). So I guess there was still enough benefit for us. However my lesson learned is that if I ever get into a similar situation, I will insist on the "getting taught" bit and not so much on productivity.

Struggling through the Jungle

The mantra we kept hearing from our boss was "Mobile First". We liked that idea, because it forced you to reduce functionality to the absolutely necessary, which leads to a lean and efficient product and a good user experience. We designed everything up from the simple.

When forging our designs in code however we ran into more and more problems which rooted, as we realised half through the project, in the base we got at ESPI. The arrangement of DOM elements got increasingly harder to manage with Backbone views and we started wondering if we did something wrong. A lot of the CSS rules didn't make sense to us either. So what happened?

At the beginning of the project we were kind of overwhelmed with all the new stuff thrown at us at ESPI. So we took a lot of stuff for granted just to keep up the pace. We didn't knew better and I think we didn't clearly enough stated the direction of the app. So the ESPI guys drew their own conclusions - which were perfectly fine at that point in the project - and we ended up with an app structure which wouldn't fit to the way we wanted the app to behave. To fix it we spent a week and basically restructured / redesigned all templates within a week. A good experience since we learned a lot about project structure and arranging stuff with handlebars and backbone.js.

Against all odds

From half way through the project we really took up speed and got good traction. We constantly increased our productivity and quality - despite all the little things we had to struggle against outside the box of our office. A loved one got terribly sick for almost three months, half of the core team only worked half time on the project, half of the Sprint planning / review meetings had to be conducted remotely via Google Hangouts or Skype - just to name a few.

The outsider might think we didn't achieve much in the last six months. But I think we did a damn good job. I think other teams - under the same circumstances - would just have plainly failed. So that's something we can be truly proud of. Working with all my talented and passionate colleagues was just a blast and I love how Open Patent Data turned out. And I am eagerly looking forward to what there's still to come for the future of patent information.

Author: Michael Kuß

OPD is a great tool to search for patents online. But no search without data. As described in one of our last post, we import the famous docdb database from the european patent office (EPO).

Since the data in the docdb are only textual information and the most interesting and funny part of a patent is an image (sort of), we show also this in our application. But how to get them?

The harvester

The most straight forward way to get the facsimile data from a patent is with the OPS API provided by the EPO. But you have to respect the fair use policy set by the EPO. So getting all documents for every week is not an option.

The best thing is to do the harvesting on demand. To get the best experience for the enduser we get the facsimile images in the format PNG from the EPO if:

a query is triggered (although if the load is too much we will drop requests for some documents)

a document is added to a folder

a document is viewed

This sound in theory quite easy. But behind the scene there are some things that are going on.

OPD will put a harvest request in a queue, namely an AWS SQS queue. This queue will feed a bee hive of small AWS EC2 instances (t1.micro) that will get the facsimiles from OPS. The retrieved images are than converted to PNG for full size view and to thumbnails shown in the patent document view. Also some metadata about the different sections of a document (bibliograhic data, drawings, description, claims, ...) are stored in a AWS DynamoDb database in order to just show the drawings in the patent.

Since the processing of one document can take some time we created two queues. One is processing the metadata and the first 3 drawings of a document. The second is processing the rest of the document. Therefore if you see a document where only the first three drawings are shown and the rest shows our placeholder, please be patient, the rest of the document is processed for you and just takes a moment.

Getting data from other sources

Of course getting all data from OPS is not the best solution. We thought about what we can do and came up with the idea to use the data already present for our running profil system. These sources can be used to prefill our databases.

We get these data as DVD as:

DEPAROM ACT

DEPAROM U

ESPACE EP (A&B)

Getting started with german data

Since we have data from the german patent office (DPMA) already in the cloud for sale we started with these to process. As a nice sideeffect we now have the complete documents since 2012 in our database (color PNG plus fulltext searchable PDF).

The general process to get this data into OPD is sketched in the following diagram:

The biggest issue about importing 2 years of data is always the processing power. Therefore we created a nice scaling tool to import the data from our shop system to amazon AWS. We get all our data from the PDFs provided by our shop.

The complete process of importing one week is as follows:

Get a import request from an SQS queue

Download the requested week from our shop

Unzip the download and push store request in a local queue that is only processed of the same ec2 instance

Get a single PDF store request from the queue and push it to s3 and push a convert request for the pdf in a general queue that is the same for all ec2 instances for the import

Get a convert request form the queue, get the PDF from s3, convert the PDF to fullsize PNGs and thumbnail and provide metadata as described above and store all in our databases

Since we have "public" queue for processing each PDF to PNG we can pretty easily scale the import on this queue. We do that also because this tasks is the most expensive one. The trick is to split down the import task to the smallest reasonable parts and call these parts from queues. This process is already pretty good known from the JEE specification.

Getting data from the EPO

After we had a running import for german patent data it was a easy feat to enhance the procedure to the EPAB-DVD data of the EPO. The only problem was to push all the data to our online store. This is implemented the same way as the german documents. Just reading metadata from the PDF is a little different from the german DEPAROM data. This source is now automaticly processed on a weekly bases the same as the germand DEPAROM data.

A real import of german and european patent data into OPD

To get data into the system we decided to import the three DVD sources DEPAROM ACT, DEPAROM U, ESPACE EP from 2012 to now into OPD. For that we started a whole bunch of ec2 instances to import. The process can be seen in the following graphics.

CloudWatch

Using Amazon CloudWatch we can have a look at our processing queues. In the graph we see how the pdf to image converter queue gets filled up by our ec2 instances. The same instances are also converting the PDFs to images. The blue line is the number of documents currently waiting to be processed and the green one shows the number of messages processed in a specific timeframe. The large drop on the right is where we started 50 instead of 10 ec2 instances to do the job. Thus improving the performance. But you also have to look at other things as seen in the next 2 sections

DynamoDb

Amazon DynamoDb is used to store metadata and thumbnails for OPD. The database has read and write capacities that you have to specify. Finding the correct values can sometime be a little bit like throwing a coin in the air. As you can see on the red line we tried to figure out how to set the correct write limit. The blue line shows the consumed write capacity. As we started to use more and more ec2 instances the write throughput growed. So we had to add some more write capacity to our database in order to keep our ec2 instances busy.

EC2 Instances

You have to get a feeling how to run the real beast. So we monitored our running ec2 instances and looked for performance drops and errors. One of the issues is to keep the things busy to not loose too much money. Here we see the graph of 50 ec2 instances that were started as seen above at some time. They started with dropped performance. So as seen above with DynamoDb we increased the write capacity to keep the ec2 instances busy at 90% CPU load. You just have to make sure that all instances are switched off again after you finished your tasks

Roadmap

We do not want to have a look at all the stuff all the time. In order to automate some stuff we will implement some rules with the Autoscaling Framework from Amazon.

Postscript

Wonder how we managed to start so many ec2 instances?

This is done by a small script we created to start a specified number of ec2 instances with a specified job. We have a number of job templates to do things on ec2.

All is based on one operation system image (AMI) that is configured with userscripts. Our script handles the correct creation of the userscript.

Author: Alexander Fetke

When we first started http://www.openpatentdata.com we already had a lot of experience regarding handling large amounts of data sets inside and outside of the cloud.  Nevertheless, the requirements of each project differ, and so we learned a lot this time as well.

Let me first present a draft of the situation.

Commitments

We committed on using bibliographic data as provided by the EPO in DocDB format.  We made this single-source decision because we were already using a multitude of sources in the Deparom Profil project and knew from experience that this always meant a lot of ongoing work due to changes in the format of the sources--which happened more often than we would have liked.

We committed on using image data for the document drawings as provided by the EPO via their OPS interfaces.  The advantage of this was that the data is for free and online available; the disadvantage was that it comes slowly, and access is restricted by a fair-use policy.  Basically this means we cannot harvest drawing images for all patent documents we provide, so we harvest the images only for the documents which at least one user seems to be interested in. (*This is likely to change soon.)

We committed not to provide any full text of the documents (yet), so all we handle are bibliographic data and drawing images.

Task Structure

Based on these commitments we set up our basic task structure:

Import the DocDB information as soon as they arrive (online via a paid download site).

Harvest the drawing image information on demand when the document becomes interesting due to user interaction.

This blog post will be about importing the DocDB information.

Importing DocDB Data

The EPO provides each week several dozens up to hundreds of XML files with new (or amended) bibliographic data of patent documents.  These XML files are compressed using zip, then the .zip files are packaged--also using zip--together with some other information like table-of-contents, DTD files, etc.  While the XML files vary in sizes of around 100MB at most, the ZIP packages typically have a size of up to 2.2GB.  For one week, this sometimes sums up to 6 or 8 GB, the whole year 2012 summed up to around 50GB.  And the numbers are increasing from year to year.

Our goal was to store each document's bibliographic data in various AWS DBs (S3, DynamoDB, and CloudSearch) as a single JSON value.

So the task was clear:  Extract the information of the DocDB packages and store it into the AWS cloud.  Since source (the packages) and target (AWS) were available online, it was feasible to do this in the cloud, using EC2 instances.

First Approach

In our first approach the EC2 instances working as importers

got their specific task (the URL of the package) from an SQS queue,

downloaded the package,

unpackaged the outer ZIP files,

unpackaged the inner ZIP files,

parsed the XMLs therein,

converted the document information properly to JSON, and

stored the values in AWS (S3, DynamoDB).

Using this setup, it was easy to have more than one importer instance work on the SQS queue, thus distributing the load.

It turned out to be a problem that a single package could be large.  The way SQS queues are designed, they are fit for relatively small tasks which can be handled in at most eight hours.  SQS queues are designed for failure; that means they work like this:  Workers get their task from the queue and state during getting how long they will need for the task at most.  This maximum duration can be at most eight hours.  When given to a worker, a task gets invisible in the SQS queue.  If and when a worker manages to finish processing its task, it informs the SQS queue about this success, then the (invisible) task is finally permanently removed from the queue.  But if the worker (for whatever reason) fails to signal success in time, the invisible task gets visible again, and any other worker might get it as well.

By this concept, a worker who does not finish its task successfully (i. e. who fails) will be backed up by a colleague after the maximum duration.  The whole system needs to be designed accordingly; it must tolerate that more than one worker might process the same task, even concurrently.

In our situation, handling one package could take longer than eight hours, at least in some cases.  With SQS queues in the standard fashion this was an unsolvable problem because no worker would ever finish the task in time, thus the message on the queue would never be removed from the queue permanently.  To solve the problem we understood that we needed to split up the packages.

Since the paid EPO download site would not provide permanent access to the packages anyway, we decided to split the process into two stages:  Storing and Importing

Second Approach

The first process in the second approach is called storing.  It means to unpackage an existing package, extract all XML files therein (and compress them using gzip because gzip can be streamed) and store them in S3 (as .xml.gzfiles).  Even with the largest packages provided by the EPO, this can easily be achieved in less than eight hours.  So we were in the parameter range of the SQS queues again.

The second process is called importing.  It now only works on single XML files which are downloaded (and uncompressed) from S3.  Since we now only have to handle a single XML file and not a package of hundreds of them in one task, the eight hour limit doesn't pose a problem anymore at this step as well.

Scheduling

When importing more than one week, we needed to ensure that all XMLs from one week had to be completed before the next week was started.  This is necessary because the EPO might issue corrections (i. e. basically corrected repetitions) of documents.  In order to always have the newest version in the DBs, the weeks must be processed in order.

To ensure this, we can use the massively parallel workers only for one week at the time.  The XMLs which take longest determine the time used for one week.  Only when all are done, the next can be scheduled.

For this, we have a special scheduler facility which creates the tasks for the XMLs of one week for all targets (S3, DynamoDB, CloudSearch, thus three tasks for each XML).  Then it waits until these tasks are completed, then it schedules the next week.

Further Thoughts

We found that S3 is much slower than DynamoDB and CloudSearch.  It posed to be the clutch in the system which resulted in each week taking about two to three hours to be imported (depending on size).  Without S3, the imports went much faster (in about one hour).  And of course, the need to import everything in order only applied to each target for its own.  We can start importing week n+1 on DynamoDB if week n has been completed on DynamoDB before; it doesn't matter if week n was completed on S3 beforehand.

There are lots of ways to enhance the system taking this new thought into account.  The way we chose was to schedule the slow target by itself, so there is a scheduler for S3 alone, and one for the other two DBs.

Currently we imported just the last three months (from week 2012/45 to the present) into our system; we also have all the data dating back to centuries past.  Importing these can be achieved later by walking backwards in time instead of forwards.  Instead of blindly overwriting existing data, we then will only store the data if there is not yet anything for the specific document (and otherwise ignore the outdated document).

Error Handling

A large amount of work went into a proper handling of all the unpredicted situations like errors occurring during parsing of the XML sources, converting it to our JSON version, etc.  The normal way of handling errors (raising an exception which leads to an abort of the program with a proper stack trace to easy finding the error) is not feasible when handling millions of documents on a farm of workers.  Simple aborts would lead to worker instances idling around (producing cost but nothing else) and not marking a task as done on the corresponding SQS queue.  After its timeout, such a task would be re-given to another worker; if the error was reproducible, it would eventually abort all workers, thus effectively stopping the whole show.

Exceptions can be handled and traces dumped to avoid aborts.  A simple try/except in the body of the specific loop does this.  It is normally not a tricky task; you just dump the current stack trace into a log file and add (hopefully) enough context information to give the developer a clue what might have raised the problem or at least to allow a reproduction of the problem in a debug environment.  But we handle bulk data in gigabyte amounts and expect to look at the collected problems only after millions of documents have been processed.

It took us a while to set up our error handling mechanisms in a way so that even after days of produced log files on dozens of instances we could still figure out which input file at which task was the trigger for the exception.  The main task in this was to reduce the size of the log files to the necessary minimum (to avoid having gigabytes of log files to analyze) without losing vital information.

Maybe you where wondering, what we are doing since the last OPD update got to you a couple weeks ago? Here is what happend:

First, you'll get a look into our staging environment that we will publish very soon to our production environment. 

Very soon means Monday, 18th of March 2013 5pm is the day and time of the launch of our official Open Patent Data Version 1.0

Please get ready to check out all the new features we developed during the last weeks, the latest design, and give it a real try.

After six month of development our vision becomes finally real.

  With OPD we provide a new way of finding, collecting and sharing patent information. And we think that we are a little bit ahead of our times with what we achieved! But much more we hope, that you will give it another try next week ...

In this update I want to take the chance, to thank my amazing team of engineers and designers. This team made my dream come true, worked and learned so hard together.

For you as an early stage user we give you the very best of what we are and could do! Hopefully you like OPD and you start using OPD in your daily business. If you enjoy to use OPD, spread a word about it.