Qippo Corporate Blog

Our recent work allowed us to create an important tool for products analysis. Now we can retrieve properties and values from a general product description text.

One of the main problems of dealing with properties is the lack of them. The product often doesn't have a specification with property name and values; it has only a description.

For example, shoes have a number of properties like Brand, Heel Height, Heel Type, Show Material, and Style that could make a search much easier. But instead, the product has only text: “These sequin covered court shoes from John Lewis will catch the light, and the playful peep toe front and cutaway sides will catch people's attention. The stiletto heels are 9cm high, providing a noticeable but not impractical lift." This text contains the necessary properties and has a lot of information about the product, but it can't be used for search or comparison, because only a person can understand it but not a computer.

It is a bit easier to deal with electronics, because the properties of some products are well specified in a table. They still need to be identified, but we can do it. This process was described in the previous posts. But the matter is that many products contain only a description as well. We have to learn how to structure descriptions and retrieve properties from them.

We succeeded in retrieving products properties directly from their descriptions. Manufacturer, color, screen size, speed, and any other category-specific properties – if they are in the description, we can retrieve them, standardize them, and use them for search.

Many people who bought laptops understand the phrase “HP 15” 2.5Ghz 512MB 300GB 4GB”. They know that 300 GB refers to hard disc capacity, and 4 GB is memory size, despite the fact that they both are measured in gigabytes. Computer won't be able to understand it automatically; we need to teach it. It has to analyze a great number of products and find out which values are used for this or that property more often. Then it has to assume that 300 GB is most likely to be hard disc capacity and not memory size, and 15" is а screen size and not a laptop width.

People write texts so that other people could read them. In order to teach a machine to do it, we have to do preliminary work first. Many things, such as statistics of values frequency and values combinations frequency, of course, can be calculated automatically. But some of the work has to be done manually. We develop each of the 1800 categories separately and prepare a special properties recognition model for each of them. We have developed a unique methodology and algorithms to teach the system to recognize any properties in any texts within a few hours.

This technology allows you to retrieve properties not only from products descriptions but from search queries as well, for example.

As always, you can test the performance of our system on your own - we have created a special demo stand for it:

http://examples.qippo.com/ominer

Qippo has 1800 categories. This is a huge number. When we collect data from different sources, one of the major issues we face is the problem of categorization. We need to find out which category the product belongs to, but how to do it? All stores have their own sets of categories, which sometimes have a lot of differences. And we need to define a catalogue for our structure. This task, which is not always obvious even for a person, becomes a real challenge for a computer.

 In order to teach a machine to identify the categories automatically, we have to create a training data set, that is, we have to allocate a vast number of products manually.  Moreover, we have to observe several rules:

Products should be distributed evenly by categories; 

The products within one category should be as diverse as possible (they should have different descriptions, properties, and so on). All possible variants of products descriptions within one category should be included.

In other words, the training set should contain the products about which the model has the least knowledge. Therefore, we add the most needed products. Qippo has a system of conflicts and their priorities as well as the products categories allocation interface implemented in it.  

It took us much time to create the training set. More than 30 moderators worked on products categorization, and they created over 1,000,000 confirmed products of great quality.

Of course, the mere existence of confirmed data won't solve the problem of categorization. It is necessary to develop algorithms that would be able to identify the essential characteristics of the objects, to find correlations and then to learn how to identify the only one correct category out of almost 2,000 categories.

We, of course, won't reveal all the secrets of how the algorithm works. We'll only mention that we have been working on it for a long time, and that the algorithm analyses the object from multiple aspects:

Product name and description analysis, identification of key characteristics for certain categories

Analysis of product properties and product properties values

Identification of differences between similar categories

Prices distribution

Image analysis

Store features

As always, we have created a demo stand where you can test the categorization quality of Qippo algorithms:

Dealing with properties

The problem of unifying properties is one of the reasons why e-stores and price comparison engines encounter different obstacles when implementing a search by properties. Different products have different set of properties or often don’t have it at all.

The problem has two parts:  

• Lack of universal specification formats. We’ll talk about it today. 

• Lack of product specification as such. We’ll discuss this interesting issue in one of our next posts. 

Lack of universal specification formats

There are no universal rules according to which manufacturers would provide their specifications. That is why one and the same property can be labeled ‘Installed Memory’ in one of the product descriptions and ‘RAM’ in the other. It can be indicated in gigabytes in one place and megabytes in others.

 A person can easily understand that they refer to the same thing, but for a computer this task is much more complicated, especially if it comes to tens of thousands of different properties and hundreds of thousands of their different variants. If a system cannot identify properties it will be impossible to implement a search.  

When dealing with properties we have to solve the problem of properties identification as well as the problem of values identification.

Properties identification

One of the problems that Qippo successfully solves is understanding that differently named properties are actually one and the same property.

We solve it by analyzing the three data components:

• Property name. We created a large library of different variants of one and the same property. For example, HDD, Drive Capacity, Internal Storage, Hard Drive Size, Hard Disk Space, SSD Spec – are different names of the same property. 

• Possibility that property belongs to a specific category. We know the typical properties of the objects of this category, so we can assume that this object has one of them.

• Property value. It is obvious, that mobile phone battery life cannot be measured in years, and laptop display size cannot be 1/2 inches.

Using the algorithms that we developed on the basis of the above rules, we determine whether the target product property belongs to one of the known properties.

In addition, Qippo has a conflicts system implemented in it. We’ll come back to it many times in our next posts. Conflicts are cases when the system was unable to automatically take a necessary decision with reasonable confidence. These cases are given to moderators for a manual analysis and then used for algorithms training. Therefore, the quality of automatic recognition constantly increases. 

Values Identification

We have to identify not only the property name but its value as well. As properties can be of different types, there are 3 ways to identify their values: 

• Numerical value. When we determine the numerical value, we keep in mind the following:  

It is necessary to determine the value metrics. Here we can come across a range of options: it is possible to write MBytes, MB, MegaBytes, and even mBytes that according to all rules should mean millibytes. The last metric unit isn’t used often and we should take it into account.   

A property has a default metric, in which it is measured for all goods in this category. We need to convert our value to the corresponding metric. For example, the property ‘Drive Capacity’ is measured in GB in this category, and we got the value 1 TB for drive capacity. So we have to convert this value to GB and write it as 1024GB.  If you want to better understand how metrics work in the system, how they are converted, and which properties have this or that value more often, you can play with our demo stand:

http://examples.qippo.com/metrics  

• Ranges, negative values. There are properties that don’t have one numerical value; they have a value range instead. And some properties can have negative values. The system, of course, should take into account both of them.  

• Enumeration value. The enumeration can be pre-defined, for example, the lists of manufacturers, colors, or processor types. The system should understand, that HP and Hewlett Packard mean the same. So we need to have a database of all possible synonyms for all similar properties and values. 

Understanding Qippo starts with its’ very important component – crawling. How do goods get into our system? How do we know how much they cost? How do we check whether products you seek are in stock? We are going to cover all of the above in this article.

Product Information Gathering Methods

All the information about products we have we get from merchant websites. There are several ways we gather such information:

Data feeds, supplied by merchants;

API, also supplied by retailers;

Crawling techniques, gathering information from individual websites.

Most price comparison engines use the first two methods above. These methods, however, have certain limitations:

Usually e-stores supply very limited information about goods they sell;

Gathering information about goods and services is typically limited to online stores that participate in paid advertising and affiliate programs. Therefore, retailers that may offer better prices and service may be left “abroad”.

When you search for sites in Google you want to find the best match possible, not the easiest or most profitable to the search engine. When you look for an online retailer you want the best one, not the one that paid for a listing. This is the exact reason why we have created template builder and Qippo-crawler in addition to the two traditional methods. 

Crawling

Here at Qippo, we select stores that we want to work with. Typically, those are the best retailers in the industry no matter whether they have data feeds or pay for link placements. 

As far as content scanning is concerned, our crawler is very similar to the ones that search engines like Google use. In particular, Qippo-crawler:

Gently treats websites, not overloading them;

Considers directives set by robots.txt and meta-tags, allowing or disallowing indexation of certain web pages.

Online stores have various types of webpages. Such pages include catalogues, forums and much other useless information. Downloading and processing such pages may be both, expensive and time consuming. As crawler scans retailer webpages, it learns to focus on pages that contain product information only. It allows scanning sites faster, as well as decreasing the server load.

There are about a thousand crawlers, working simultaneously in Qippo. Their work is supervised from a special section in the administrator panel. Here is how it looks like for a particular store:

   We constantly monitor crawling quality and automatically sort out potential problems. Information about such problems is constantly gathered and analyzed. 

Template Building

Once we get a page from a particular store a question arises: how may a system find out where the product information is located, such as name, price, availability and other data? 

There are 3 practical solutions here:

An analyzer that can recognize data on a webpage using certain markers. Unfortunately, this method is not error-proof and is not capable of gathering all necessary information.  

Using micro data tags on a webpage (schema.org/Product). It is an amazingly convenient technology, supported by all leading search engines. Unfortunately, the number of websites, using this technology is miniscule.   

Creating “recognition rules” for pages. Each store is assigned its own individual template with data extraction rules. Such method allows receiving needed information with no room for mistake. The downfall is that a unique template should be created for each individual store, making it a time-consuming endeavor, requiring extensive knowledge of HTML. 

Qippo requires precision along with high volume, and the first two methods are too limited to accomplish it. Therefore, the third method is a fit. However, creating individual templates for tens of thousands of stores is almost an impossible task to complete. 

We have solved this problem by creating a template generator. It is based upon data mining technology created by Web Harvest (web-harvest.sourceforge.net), and our generator just creates individual code for this platform.

It all looks very simple: you set several page examples for the template generator, then click on page areas with information about product name, pricing and so forth and press “Save”. That’s it! The site is ready to be crawled! Such template creation rarely takes more than 10 minutes. 

For more sophisticated product pages, containing databases there is a special mode “Name=>Value”, allowing data extraction from tables.

To show you how it all works we have created a demo store based on Spree engine:

spree-stand.qippo.com

and made a 37 second video of how a template is created for this demo store:

You may check how the crawler works for the template built above. Go to the “Examples” page:

examples.qippo.com

We are a dedicated team of developers with one idea in mind – to make E-Commerce user-friendly and exciting for our clients.

We are thrilled about the way Internet have changed our lives and how easy and convenient it is today to buy virtually everything online.

A choice of millions of goods from thousands of stores makes you feel both, overwhelmingly thrilled about your options and sarcastically quizzed about making the right choice.

That is exactly why a number of online services emerged, allowing you to compare online retailers. Yet, Qippo is different from your average Price Comparison Engine.

We have set a goal to create a comprehensive consumer tool, integrating all aspects of comfort shopping – a must-have instrument that would:

Assist as a friendly salesperson: listen to your preferences, offer suitable choices and answer any questions you may have, as sometimes you don’t exactly know what you want but only have a general idea about a product. It is our job to understand your expectations and desires and offer you the best solution.  

Act as a good friend, having different opinion about that coffeemaker or bicycle you are about to buy. Internet is full of product reviews and ratings, but can you really comprehend them all? And, what is more important, which review would you trust most?  

Perform as a market analyst, having an objective expert opinion. Say, you have a crush on a certain laptop. Should you go with it or wait for a newer release? Should you consider another maker with a cheaper but comparable model?  

Serve as a personal assistant: find the best store for your demand; not the one with flashy advertising or some glamour magazine “Store of the Year” rating with overinflated prices, but a reliable honest seller offering the best deal around.

To bring you all of above we have developed a number of amazing and truly helpful solutions.

Some of them are already implemented in the website, while the others are yet to be released for you to enjoy.

We are going to share all of our solutions in this blog:

How do we search products on the web? 

How difficult is it to use consumer reviews?

How to find product characteristics, even if they are not listed?

How consumer goods are categorized and merged?

How do we know what is out there for you?