OrientDB's development lead blog

Hi all,

Since 2.2 version we support incremental backup feature. So what perks do you get if you use incremental backup instead of classical backup :

Database stores on disk only data which were changed after previous backup.

Database is not switched to read only mode and users may continue to work with it.

Lets look how does it work inside of OrientDB.

Every time when we change data, together with user data we put some kind of timestamp which will show us when this change was happend, we call it LSN. Despite normal timestamp, every time, when we make update, LSN will continuously grow and can not be equal to previous LSN. In each snippet of incremental backup (file which is created during single incremental backup operation) we put maximum LSN of changes which we added into this snippet. So merge of different snippets and finding of changes happened after previous backup is kinda simple. We iterate over all data in sequential way and compare latest stored LSN and LSN of processed change, if last one is bigger we put it in new backup file.

To prevent situation when we lose changes happened during iteration over all database data, we log all operation since start of incremental backup process into database journal and at the end of backup process we append list of those operations to the backup file.

During restore process we import all data added into backup files and apply operations from database journal, so at the end of the process we get database in the state which it had at the end of last incremental backup.

To perform incremental backup from Java you can call method ODatabase#incrementalBackup(path-to-incremental-backup-directory) or from console the same will look like: backup database path-to-incremental-backup-directory -incremental .

To create database from incremental backup you can call from Java ODatabase#create(path-to-incremental-backup-directory) or from console: create database root root plocal graph -restore=path-to-incremental-backup-directory

Every user who works with OrientDB knows that to manipulate records he or she has to create ODatabaseDocumetTX instance first. But there is  wide misunderstanding about meaning of this object. Many users think that this object may be treated as abstraction of data which are stored on disk and placed on server. As result they create single instance of this object per thread, assign it to the class field and use the same object during whole application life. That is "not performance" wise decision. There are several reason why you should consider to use short living database instance acquired from the pool for example from OPartitionedDatabasePool instead.

The most important reason is that during loading of records we put records into local cache which is based on WeakHashMap.  We need this cache for 2 reasons:

To avoid OConcurrentModificationException in case of loading of the same record in call stack.

To speed up graph traversal.

Lets look how WeakHashMap works (in OpenJDK at least):

Every put/get/remove method of this HashMap calls getTable() method.

Which in turn calls expungeStaleEntries() method.

Responsibility of last method is to remove weak references which are already not accessible. expungeStaleEntries() uses ReferenceQueue which was passed during creation of WeakReference object to detect unreachable references.

The main problem there is mechanics which is used to fill in and pull items from  reference queue. When weak reference is becoming a subject of garbage collection special thread which has high priority java.lang.ref.Reference.ReferenceHandler put such reference into reference queue. But reference queue itself is guarded by object wide lock !

Let's put all of this together:

You have long living ODatabaseDocumetTX instance.

You use it to load many short living record objects.

You have WeakHashMap polluted by many WeakReferences.

After next GC run ReferenceHandler starts to pull and lock ReferenceQueue and as result lock WeakHashMap objects.

Your threads are become frozen for a long time. We know situations when threads were frozen for several seconds !

So main rule of thumb - never use long living ODatabaseDocumetTX instances, use database pool instead. Despite of solving of problem described above pool also provides support for nested transactions.

We are looking at driver developers for OrientDB database (http://www.orientdb.com) with good knowledge of at least 2 of the following languages: PHP, Python and C#(.NET platform). The candidate will learn the missing language and will contribute in the long term to the development/support of other drivers, like Golang.

Good knowledge of TCP/IP and HTTP protocols is a big bonus. In short term perspective this developer will analyse existing community drivers, fix existing issues and implement new API together with the OrientDB core team.

Furthermore, the candidate has to stay updated with the new trending frameworks and evaluate what to support as a new connector for OrientDB.

So in nutshell wer are looking for developers with the insane passion for the polyglot programming and studying of the next cool framework.

News itself is written in blog title, but in blog itself I want to clarify some details of implementation. To do not loose disk cache statistics of frequently used data and as result to do not slow down user queries we store whole statistic inside of file with name cache.stt  and then load those data when database is opened for the first time. Which means that users will experience slow down during server shutdown and during the cases database is opened for the first time. 

If you want to avoid slow down during database open operation you may indicate that database should be opened during the server start. Simply add attribute loaded-at-startup=true in orientdb-server-config.xml->storages->storage tag. It may look like following: <storage name="db"         path="plocal:${ORIENTDB_HOME}/db"  loaded-at-startup="true"/>

Many users ask how much of memory will be consumed by OrientDB and which settings affects this number. This question becomes even more actual since 2.2 release. In new release we allocate memory by big chunks (about of 1gb size each) and then split it between threads on demand. 

There are 2 reasons why we have chosen given approach:

Fix OS allocation issues like this one

Prepare to the change when we will assign adjacent memory chunks for each thread on demand. I mean following case. Let suppose first thread got the page with address 0x1 but second thread got the page with address 0xFE. Then on next request for new page first thread will get page with address 0x2 and second thread will get page with address 0xFF. We hope It will allow us to improve system performance because of better usage of CPU cache locality feature.

So how to calculate maximum amount of memory which will be consumed by OrientDB. That is simple. OrientDB uses both heap and direct memory. Direct memory is used in disk cache and database journal. 

Memory consumed by disk cache may be calculated looking on value of following configuration parameter storage.diskCache.bufferSize which shows maximum amount of memory consumed by disk cache in megabytes.

You may get maximum amount of pages consumed by database journal by reading value of storage.wal.cacheSize  parameter. You should increase this value by 1 and multiply on value of parameter storage.diskCache.pageSize . In this way you will get maximum amount of direct memory consumed by database journal.  (Since 2.2-beta 2 WAL buffer uses heap not  direct memory at sould not participate in calculation of direct memory consumption).

After that we need sum both calculated values and increase result till it will not be divided to the 1GB without reminder (size of chunk of memory allocated by pool at once). 

The rest is simple add value which you calculated above and add amount of memory consumed by heap and you will get maximum amount of memory which will be consumed by OrientDB.

UPDATE 1: Maximum size of memory chunk in bytes which will be allocated by OrientDB may be set using property memory.chunk.size .

UPDATE  2:  Since 2.2-beta 2 WAL buffer uses heap not  direct memory at sould not participate in calculation of direct memory consumption.

Since the creation of plocal storage OrientDB used Unsafe (and JNA on distributions where Unsafe API was incompatible with Oracle JDK Unsafe API) to implement 2 features:

Direct memory buffers

Fast serialization of data

And if it is not completely possible yet to remove usages of Unsafe in implementation of second feature, we decided to completely migrate our direct memory implementation to use ByteBuffers instead of Unsafe. From the first glance it seems like usage of ByteBuffer is not justified because of:

Call to Unsafe thanks to JVM intrinsic will be converted to single CPU command

Memory allocated in ByteBuffer cannot be deallocated

Well that is true but …

When we use Unsafe we always make range checks to avoid JVM crash and track current position of data which we are going to write or read from page.

If we compare overall system complexity which includes deserialization of records, calling of hooks, complexity of data structure algorithms and do not forget JVM optimizations, cost of call of ByteBuffer methods is not so high.

We do not need to deallocate memory in ByteBuffer, we merely put ByteBuffer in pool.

In FileChannel we call methods which directly use DirectByteBuffer and as result avoid unnecessary copying of data between heap and direct memory.

Taking all of this in account and running several benchmarks we found out that after migration to ByteBuffer in some cases (for example in case of full scan when not all of the data are preallocated in disk buffer) database operations show not only the same performance numbers but even made faster!

There is one disadvantage of usage of such approach if you work with embedded database you have to set value of -XX:MaxDirectMemorySize option to the amount of RAM available on your computer. In case of server installation, we set this value in server startup script to the 512 GB which should be enough for many cases.

In 2.2 version of OrientDB we have added tools which allow to gather storage performance metrics for whole system and for concrete command which is executed at the current moment.This feature will be more interesting for database support team but probably it will be also interesting  for users who want to understand why database is fast or slow for their use case and what is the reason of numbers which they get during benchmark.

But before we will consider characteristics which are gathered during profiling of storage lets look a bit on OrientDB architecture.

All high level OrientDB components which exposed to the user like cluster or index are implemented inside of storage engine as “durable components” and extend ODurableComponent class. This class is part of framework which was created to make components/data structures operations atomic and isolated in terms of  ACID properties. Each durable component have to hold it’s data in direct memory , not in Java heap. But if in Java we operate variables to store/read application data, durable component operates pages. Page is continues snippet of direct memory which always have the same fixed size and is mapped to file placed on disk. When data is written to the page they automatically written to the file, but data are not written instantly, some time have to pass between moment when data is written to the page and time when data will be written to the file.

We separate write operations on pages and on file system because file system operations are slow and we try to decouple data operations and file system operations. When we change page it is not written to the disk instantly as I have already mentioned above but is placed in write cache. Write cache aggregates all changed pages  and stores them to the disk in background thread in order of their position in file. So if we have changed pages with positions 3, 2, 8, 4, they will be stored in order 2, 3, 4, 8. Pages are sorted by their positions in file because does not matter whether you use DDR, SSD or HDD to store your data, sequential IO operations are always faster than random IO operations. Because pages are stored in separate background thread, speed of disk write operations will be decoupled from speed of data modification operations. 

In case of write operations we may delay write of data and try to convert them in sequential IO operations, but if we need to read data we need them instantly and can not delay read of data from file. So for this case we use well known technique of caching of frequently used pages in read cache.

So taking all above into account you can see that OrientDB uses 2  caches:

Read cache - to cache frequently used pages.

Write cache - to decouple data modification operations from file system write operations and to make those operations sequential.

When we read page from file following steps are performed:

Read page from read cache , if page is found we quit there.

Read page from write cache, if page is found in write queue we quit there.

Read page from file.

Add page to the read cache and return found page.

When we modified page content it is automatically placed in write cache.

There is one big problem with all those caches. Such system is not durable. If application is crashed then some data which have not been written yet to the disk will be lost. 

To avoid this kind of problems we use database journal aka WAL (write ahead log). Which makes whole process of writing of data a bit complex. When we modify page we do not put page to the write cache , instead of that we write diff of page content into map  keys of which consist of file and index of changed page and values of which contain diff of changes between original and changed page.

When operation on cluster or index is completed without exceptions we extract all changes from the map and log them inside of database journal and only after that we apply those changes to the file pages and put them to the write cache. Database journal may be treated as append only log so all write operations to the database journal are sequential and as result are fast. Process of writing of changes to the database journal and applying them to the “real” file pages is called “atomic operation commit”.

What value database journal gives to us ? 

If process is crashed in the middle of atomic operation (before atomic operation commit) nothing bad will happen because “real” data are not changed at this moment.

If process is crashed during atomic operation commit we will restore this operation on next  start of database from database journal, if database journal contains only half of operation this operation will be merely not restored.

In both cases data consistency will not be compromised.

Taking all of above into account you probably have already concluded that main performance characteristics of OrientDB storage engine (and not only OrientDB)  are:

Speed of read of pages from the cache.

Speed of write of pages to the cache.

Speed of read of pages from the file.

Speed of write of pages to the file.

Amount of pages read for single component operation.

How much time is spent on atomic operation commit.

Percent of cache hits - how many times we do not need to read data from file system but load them from the cache memory.

All those numbers will show us in which direction we have to evolve our project. For example if we have good numbers for disk cache hit rate and very few pages are read for single component operation we have to improve disk cache speed as whole, but if we have a lot of pages are read for single component operation and very low numbers of speed of read of pages we need to minimize amount of pages accessed for single operation and convert data structures to ones which use more sequential than random IO operations. 

Readers may ask: “well all of this is very good but how it is related to us” ?

The answer is: when you report performance issue please provide not only benchmark, because we all have different hardware and sometimes we merely can not reproduce your issue , but also performance numbers gathered as result of profiling of storage.

How to do that ?

It may be done in 2 ways, using JMX and using SQL commands.

JMX console provides numbers are gathered from execution of all operations in storage but SQL commands provide data which are gathered for selected set of commands.

To gather performance for selected set of commands you can for example execute following script:

At the end of the script you will see following result:

As you can see you may see numbers of storage performance as whole and numbers of performance of each component.

Data from atomic operation commit phase are presented as data from component with name “atomic operation”.

If you work with embedded database you can start and stop storage profiling by calling of methods OAbstractPaginatedStorage#startGatheringPerformanceStatisticForCurrentThread() to start storage profiling and OAbstractPaginatedStorage#completeGatheringPerformanceStatisticForCurrentThread() to stop profiling of storage.

Also you may connect to JMX server and read current performance numbers from MBean with name: com.orientechnologies.orient.core.storage.impl.local.statistic:type=OStoragePerformanceStatisticMXBean,name=,id=

Hope it will be interesting for you to read overview of OrientDB architecture and performance characteristics which are important for us. Please do not forget to send them together with performance reports. 

Since 2.1 (next hotfix) and 2.2 versions of  #OrientDB it will be possible to change size of direct memory disk cache at runtime. 

That is quite simple to do, merely call com.orientechnologies.orient.core.config.OGlobalConfiguration#DISK_CACHE_SIZE.setValue() at any place in your program and size of disk cache will be changed (passed in value is cache size in megabytes). You may do this inside of your code or using OrientDB console or Studio.

There are couple of nuances of usage of given feature:

If you increase cache size that is pretty cheap operation.

If you decrease cache size it may take some time because previously loaded pages have to be freed.