witwall

cRARk is a .rar archive password cracker, but unlike rarcrack, can be customised to a far greater extent to allow partial passwords, wordlists, complementing wordlists with characters and more. Also this programme is CUDA enabled which allows for a vast increase in testing speed. This of course only if you have a capable graphics card. (http://www.nvidia.com/object/cuda_gpus.html) cRARk is not installed by default on BT4 Final, a shame really as it is more versatile than rarcrack in my opinion. To install it though is a piece of cake ;

apt-get install crark

! During the install, the program tried to install the CUDA dependacies as well, in order for this to work properly you will need to exit 'X'. To do this press Ctrl + Del + Backspace to get back into Command Line only. So seems best way to install is to get online, then quit X with Ctrl + Del + Backspace, and then run the apt-get install from the command line interface. The machine I installed it onto does not have any CUDA capabilities, so this post will only go through some of the functions of cRARk without testing the CUDA improvements on cracking speed. I may later try to do a full HDD install on my desktop to see if I can get my 8800GTS to work. So after installing cRARk, lets fire it up through either the menu;

Start/Dragon --> Backtrack --> Privilage Escalation --> Password Attacks --> OfflineAttacks --> Crark

or command line ;

cd /pentest/passwords/crark/

Lets check out the files the readme's and what general options are advised;

ls./crark

BRUTEFORCE ATTACKS================

To start off we need to create the password.def file. To do this simply copy either the english.def file or the crackme.def file to password.def ;

cp crackme.def password.def

Then to modify the password.def file to reflect the options we want to use. Here I will be working with simple password protected files stored on my flashdrive (mounted on /media/8GB/) Testing password protected test100.rar file with password 100 ;

nano password.def

Under the double hash enter the testing methods desired, in this case as only numbers ;

[$1] *

Save and exit

As my system cannot use CUDA I will disable it with the -c command when testing.

./crark -c /media/8GB/test100.rar

For checking password protected testabc.rar file with lowercase letters only ;

nano password.def[$a] *

Save and exit.

./crark -c /media/8GB/testabc.rar

For checking a password protected test-ABC.rar with uppercase characters only ;

nano password.def[$A] *

Save and exit

./crark -c /media/8GB/test-ABC.rar

To check all of the above on password protected testaB1.rar ;

nano password.def[$a $A $1] *

Save and exit

./crark -c /media/8GB/testaB1.rar

!! It is important to note that cRARk will start checking based on the same sequence as the characters are entered in the definition file.So in the above example entering the character information in the definition file as $1 $a $A would take a lot longer to find the password then if using $a $A $1 . The end result will be the same, cRARk will find the password, but the time needed for checking will be severely affected.(I thought I had messed up somewhere earlier on when I changed character sequence in definition file and it took longer than previous attempts).

To go all out and also include special characters ;

nano password.def[$a $A $1 $!] *

Save and exit. To test the password.def file and see whether the commands are going to work OK, the programme can test run the definitions using the -v option ;

./crark -c -v /media/8GB/test100.rar

This wont actually start the cracking process, but will print the characters on screen so you can check whether its doing what you want it to. To specify the number of  characters, the options -l & -g can be used. So to specify a minimum of 3 characters and a maximum of 5 characters;

./crark -c -l3 -g5 /media/8GB/testabc.rar

The bruteforce attempts resulted in an average of around 60 pass/sec on my netbook and around 160 pass/sec on my desktop (windows version of cRARk). With CUDA this will be greatly enhanced, but have to see if/when I can get that going.

WORDLIST/DICTIONARY ATTACKS======================

cRARk also has the option to check passwords from a wordlist file. To do this the password.def file needs to be adjusted to specify the location of the wordlist with $w, in this case I have copied a small english wordlist called english.dic into cRARk's directory. (You can of course also specify an alternative location where the wordlist is)

   Then we need to specify that cRARk will use a wordlist in password.def ;

nano password.def$w = "english.dic"##$w

Save and exit.

./crark -c /media/8GB/testAmsterdam.rar

The dictionary attacks averaged around 20 pass/sec on my netbook and I understand that there is no CUDA support for the dictionary attacks.. So having a focussed wordlist is a must ! To allow an easier choice of options, it is also possible to prepare some definition files and then specify these with the -p option. For instance you can prepare a .def file which will use a wordlist file as follows and then keep there for future use ; cp password.def wordlist.def

Then to specify to use the newly created wordlist.def file, the -p option is included ;

./crark -c -pwordlist.def /media/8GB/testAmsterdam.rar

There are a huge number of options to play with when it comes to the wordlist usage, extra options can be included to capitalize / invert / mashup / add numbers etc etc etc I may update the post to show some of these options on a test wordlist. It will be interesting to see how far I can get with the CUDA side of things, however that would mean a full HDD install on the desktop (doesnt work in VMware). Author's Helpfile & Linkage ; http://www.crark.net/cRARk.html

Video on the above ; 

There are a large number of password cracking (or to word it in a nice fashion, password recovery) programs available to crack passwords of any number of file type. Here I will be looking at cracking password protected archive files with rarcrack which is included in the back|track 4 distro. First lets navigate to rarcrack in back|track, see the help file and which files are located in the rarcrack directory.

cd /pentest/passwords/rarcrack ./rarcrack --help ls

There are 3 test files included in the rarcrack directory, but lets try rarcrack on some of the files which I created which are on a USB drive; /media/4G/ Starting an attack ; This below on a zip file created with WinRar;

./rarcrack --type zip --threads 8 /media/4G/TEST6-winrar.zip

This one below on a zip file created in 7-Zip with ZipCrypto encryption; ./rarcrack --type zip --threads 8 /media/4G/TEST-ZipCrypto.zip

This one below on a 7z archive with AES256 encryption;

./rarcrack --type 7z --threads 8 /media/4G/TEST1-AES256.7z

So Slow ! When a crack attempt is started, an xml status file is created in the directory where the archive file is located. So we can stop the crack and edit the values of the xml file to help speed up the cracking process.

This xml file can be editted to change the character list being used for the crack, in this case as I know the password is a numerical value, we can edit the xml file so that rarcrack only checks numbers; nano /media/4G/TEST1-AES256.7z.xml

Changing the character set to numerical only;

Now we restart the attack on the 7z file and the attack will resume but now only check numerical values; ./rarcrack --type 7z --threads 8 /media/4G/TEST1-AES256.7z

Video on the above using rarcrack can be found here ;

http://blip.tv/file/2816224 or http://www.youtube.com/watch?v=BMFn-jps3iY Although I am trying to stick to the back|track tools in my posts, I have to divert somewhat here and mention a Windows tool by Elcomsoft; "Advanced Archive Password Recovery" (ARCHPR). It is a great tool and Elcomsoft have password recovery tools for a fantastic number of filetypes. It has an easy interface with various cracking options such as Bruteforce and Dictionary attacks, and is also able to include characters which you think may be correct and mask those you want testing, as in; pass????. The speed reached is much better in most cases than what rarcrack achieves and also is more flexible on which files can be chosen, although it does not support 7zip created archives. With rarcrack I was having trouble with it catching the passwords on zip files with AES encryption, ARCHPR has no trouble with these. ARCHPR in action with bruteforce options ;

越来越多的人开始意识到，网站即软件，而且是一种新型的软件。

这种"互联网软件"采用客户端/服务器模式，建立在分布式体系上，通过互联网通信，具有高延时（high latency）、高并发等特点。

网站开发，完全可以采用软件开发的模式。但是传统上，软件和网络是两个不同的领域，很少有交集；软件开发主要针对单机环境，网络则主要研究系统之间的通信。互联网的兴起，使得这两个领域开始融合，现在我们必须考虑，如何开发在互联网环境中使用的软件。

RESTful架构，就是目前最流行的一种互联网软件架构。它结构清晰、符合标准、易于理解、扩展方便，所以正得到越来越多网站的采用。

但是，到底什么是RESTful架构，并不是一个容易说清楚的问题。下面，我就谈谈我理解的RESTful架构。

一、起源

REST这个词，是Roy Thomas Fielding在他2000年的博士论文中提出的。

Fielding是一个非常重要的人，他是HTTP协议（1.0版和1.1版）的主要设计者、Apache服务器软件的作者之一、Apache基金会的第一任主席。所以，他的这篇论文一经发表，就引起了关注，并且立即对互联网开发产生了深远的影响。

他这样介绍论文的写作目的：

"本文研究计算机科学两大前沿----软件和网络----的交叉点。长期以来，软件研究主要关注软件设计的分类、设计方法的演化，很少客观地评估不同的设计选择对系统行为的影响。而相反地，网络研究主要关注系统之间通信行为的细节、如何改进特定通信机制的表现，常常忽视了一个事实，那就是改变应用程序的互动风格比改变互动协议，对整体表现有更大的影响。我这篇文章的写作目的，就是想在符合架构原理的前提下，理解和评估以网络为基础的应用软件的架构设计，得到一个功能强、性能好、适宜通信的架构。"

(This dissertation explores a junction on the frontiers of two research disciplines in computer science: software and networking. Software research has long been concerned with the categorization of software designs and the development of design methodologies, but has rarely been able to objectively evaluate the impact of various design choices on system behavior. Networking research, in contrast, is focused on the details of generic communication behavior between systems and improving the performance of particular communication techniques, often ignoring the fact that changing the interaction style of an application can have more impact on performance than the communication protocols used for that interaction. My work is motivated by the desire to understand and evaluate the architectural design of network-based application software through principled use of architectural constraints, thereby obtaining the functional, performance, and social properties desired of an architecture. )

二、名称

Fielding将他对互联网软件的架构原则，定名为REST，即Representational State Transfer的缩写。我对这个词组的翻译是"表现层状态转化"。

如果一个架构符合REST原则，就称它为RESTful架构。

要理解RESTful架构，最好的方法就是去理解Representational State Transfer这个词组到底是什么意思，它的每一个词代表了什么涵义。如果你把这个名称搞懂了，也就不难体会REST是一种什么样的设计。

三、资源（Resources）

REST的名称"表现层状态转化"中，省略了主语。"表现层"其实指的是"资源"（Resources）的"表现层"。

所谓"资源"，就是网络上的一个实体，或者说是网络上的一个具体信息。它可以是一段文本、一张图片、一首歌曲、一种服务，总之就是一个具体的实在。你可以用一个URI（统一资源定位符）指向它，每种资源对应一个特定的URI。要获取这个资源，访问它的URI就可以，因此URI就成了每一个资源的地址或独一无二的识别符。

所谓"上网"，就是与互联网上一系列的"资源"互动，调用它的URI。

四、表现层（Representation）

"资源"是一种信息实体，它可以有多种外在表现形式。我们把"资源"具体呈现出来的形式，叫做它的"表现层"（Representation）。

比如，文本可以用txt格式表现，也可以用HTML格式、XML格式、JSON格式表现，甚至可以采用二进制格式；图片可以用JPG格式表现，也可以用PNG格式表现。

URI只代表资源的实体，不代表它的形式。严格地说，有些网址最后的".html"后缀名是不必要的，因为这个后缀名表示格式，属于"表现层"范畴，而URI应该只代表"资源"的位置。它的具体表现形式，应该在HTTP请求的头信息中用Accept和Content-Type字段指定，这两个字段才是对"表现层"的描述。

五、状态转化（State Transfer）

访问一个网站，就代表了客户端和服务器的一个互动过程。在这个过程中，势必涉及到数据和状态的变化。

互联网通信协议HTTP协议，是一个无状态协议。这意味着，所有的状态都保存在服务器端。因此，如果客户端想要操作服务器，必须通过某种手段，让服务器端发生"状态转化"（State Transfer）。而这种转化是建立在表现层之上的，所以就是"表现层状态转化"。

客户端用到的手段，只能是HTTP协议。具体来说，就是HTTP协议里面，四个表示操作方式的动词：GET、POST、PUT、DELETE。它们分别对应四种基本操作：GET用来获取资源，POST用来新建资源（也可以用于更新资源），PUT用来更新资源，DELETE用来删除资源。

六、综述

综合上面的解释，我们总结一下什么是RESTful架构：

（1）每一个URI代表一种资源；

（2）客户端和服务器之间，传递这种资源的某种表现层；

（3）客户端通过四个HTTP动词，对服务器端资源进行操作，实现"表现层状态转化"。

七、误区

RESTful架构有一些典型的设计误区。

最常见的一种设计错误，就是URI包含动词。因为"资源"表示一种实体，所以应该是名词，URI不应该有动词，动词应该放在HTTP协议中。

举例来说，某个URI是/posts/show/1，其中show是动词，这个URI就设计错了，正确的写法应该是/posts/1，然后用GET方法表示show。

如果某些动作是HTTP动词表示不了的，你就应该把动作做成一种资源。比如网上汇款，从账户1向账户2汇款500元，错误的URI是：

POST /accounts/1/transfer/500/to/2

正确的写法是把动词transfer改成名词transaction，资源不能是动词，但是可以是一种服务：

POST /transaction HTTP/1.1 Host: 127.0.0.1 from=1&to=2&amount=500.00

另一个设计误区，就是在URI中加入版本号：

http://www.example.com/app/1.0/foo

http://www.example.com/app/1.1/foo

http://www.example.com/app/2.0/foo

因为不同的版本，可以理解成同一种资源的不同表现形式，所以应该采用同一个URI。版本号可以在HTTP请求头信息的Accept字段中进行区分（参见Versioning REST Services）：

Accept: vnd.example-com.foo+json; version=1.0

Accept: vnd.example-com.foo+json; version=1.1

Accept: vnd.example-com.foo+json; version=2.0

 Contents

The Problem with PDFs

An Introduction to Flying Saucer and iText

Generating a Simple PDF

Generating Content on the Fly

Page-Specific Features

Rendering Generic XML Instead of XHTML

Generating PDFs in a Server-Side Application

Conclusion

Resources

PDFs are one of the most common and most significant document formats on the internet. Typically, developers must use expensive tools from Adobe or cumbersome APIs to generate PDFs. In this article, you will learn how to programmatically generate PDFs easily with plain XHTML and CSS using two open source Java libraries: Flying Saucer and iText.

The Problem with PDFs

PDFs are a great technology. When Adobe created the PDF format, they had a vision for a portable document format (hence the name) that could be viewed on any computer and printed to any printer. Unlike web pages, PDFs will look exactly the same on every device, thanks to the rigorous PDF specification. And the best thing about PDFs is that the specification is open so you can generate them on the fly, using readily available open source libraries.

There is one big problem with PDFs, however: the spec is complicated and the APIs for generating PDFs tend to be cumbersome, requiring a lot of low-level coding of paragraphs and headers. More importantly, you have to use code to generate PDFs. But to make good-looking PDFs, you need a graphic designer to create the layout. Even if graphic designers are up to the task of programming, they still must convert their layout from some other format to code, which can be cumbersome, buggy, and time-consuming. Fortunately, there is a better way.

The way to make good looking PDFs is to let the programmers do what they are good at: writing code that manipulates data, and let the graphic designers do what they are good at: making attractive graphic designs. Flying Saucer and iText are tools that do this. They let you render CSS stylesheets and XHTML, either static or generated, directly to PDFs.

An Introduction to Flying Saucer and iText

Flying Saucer, which is the common name for the xhtmlrenderer project on java.net, is an LGPLed Java library on java.net originally created by me and continually developed by the java.net community. Download it from the project page, or use the copy included with this article's sample code (see Resources). Flying Saucer's primary purpose is to render spec-compliant XHTML and CSS 2.1 to the screen as a Swing component. Though it was originally intended for embedding markup into desktop applications (things like the iTunes Music Store), Flying Saucer has been extended work with iText as well. This makes it very easy to render XHTML to PDFs, as well as to images and to the screen. Flying Saucer requires Java 1.4 or higher.

iText is a PDF generation library created by Bruno Lowagie and Paulo Soares, licensed under the LGPL and the Mozilla Public License. You can download iText from its home page or use the copy in the download bundle at the end of this article (seeResources). Using the iText API, you can produce paragraphs, headers, or any other PDF feature. Since the PDF imaging model is fairly similar to Java2D's model, Flying Saucer and iText can easily work together to produce PDFs. In fact, the PDF version of the Flying Saucer user manual was itself produced using Flying Saucer and iText.

Pdf2AfpLib is a library for converting Pdf files to Afp files. It's written in C# and uses Ghostscript for a part of the conversion process. At the moment it generates only black and white Afp files. No gray scale, no color. Important: all the content is rasterized. In the resulting afp file, every page contains only one big image that completely covers the page. The conversion runs through 2 steps:

Pdf is converted to tiff G3 files using Ghostscript. One tiff for each page.

Tiff are parsed and the image content is imported as an IOCA image in the output afp.

Google Docs™  backup utility Overview GDocBackup is simple utility to backup (export) all your documents from Google Docs™ to local disk. It's open source, freeware and licensed under the Apache License Version 2.0  It does a simple backup: for each document in Google Docs, it downloads the document if the document is not present on the local disk or if it has a different date. GDocBackup runs on Windows (tested on XP, Vista and 7), Linux (tested on Ubuntu 8.10 and OpenSuse 11.1) and Mac (tested on OS X 10.5.6). Required .NET or Mono.

http://urldiff.appspot.com/ Architectural overview UrlDiffMonitor (UDF) allows users to detect changes on web pages. The user submits an Url. UDF will regularly check if there was modified content on the page (added or removed). The user will be notified of modification through Atom Feed, Email and Google Talk instant message. UDF is an application developed on top of Google App Engine (java version).

Blocks:

Web application: it allows users to manage their URL (add new url, delete url and list them) and to obtain the Atom Feed. 

Checker: it's an automated unit. It runs regularly and detect changes on remote web pages (user urls). Checker is regularly started by a Cron Service trigger. During every run it schedules check activity on the TaskQueue Service.

Feeder: it publishes the user Atom Feed. 

Steps:

User submits a new URL

URL is stored

Cron Service triggers the Checker

Checker detects which URL needs to be checked (new and "elapsed" urls)

Checker enqueues check activities for URLs of the previous step (1 check task for each url)

TaskQueue service starts the check activities, one at a time.

Checker detects changes

executes a HTTP GET

gets plain text from Html (remove tags, scripts, styles, etc.)

compares old text (retrieved from DataStore) with new text (from http get)

execute the diff algorithm

If changes are detected, Checker: 

stores information to DataStore (Last modify date, diff info, new text, etc.)

send email and xmpp notifications

A user's feed reader requests the feed (calls Feeder url)

Feeder builds the Atom xml feed using data from Datastore and Memcache.

Notes:

Authentication is based on Google Authentication (integrated in App Engine). Each user needs to have a valid Google account.

How to get the level of nested Xobjects in a pdf file using iTextSharp.

[Fabrizio Accatino - [email protected]]

Using iTextSharp (or iText)  it's possible to walk through the pdf objects. In particular, it's possible to detect the presence of a XObject, "open" it and walk through its internal objects. A pdf file is a collection of objects (texts, lines, rectangles, images, etc.) placed on its pages. A XObject is a particular object: it is container of objects. An xobject can be placed on a single page or on more pages but its content is stored once in the pdf file. Xobjects are very useful to repeat a complex and heavy content on many pages, storing it once in the pdf. An xobject can contain one or many xobject. Every contained xobjects can contains other xobject and so on. I don't know if the Pdf Reference has a limit on the maximum number of nested object levels. If you print a pdf with many nested xobject on a PostScript printer you can get a fatal error:     ERROR:limitcheck     OFFENDING COMMAND: gsave     STACK:      -savelevel- This is due to the fact that every xobject is "translated" to PostScript with a couple of gsave + grestore commands. A gsave command saves the current graphics status into the buffer (stack). The grestore restores the graphic status.  But the buffer (every buffer) is limited. When it is full, you get the error.  Tipically a postscript RIPs can manage about 15 levels of nested objcts. Using  PdfCheckXObjectLevels.Checker you can get informations about level of nested xobject in a pdf. At the moment the source code is a very very beta. (suggestions / feedback are very welcome) Usage:

PdfCheckXObjectLevels.Checker chk = new PdfCheckXObjectLevels.Checker();

chk.Exec("simple.pdf");

Console.WriteLine(chk.MaxLevel);

The class:

using System;

using System.Collections.Generic;

using System.Text;

using iTextSharp.text.pdf;

namespace PdfCheckXObjectLevels

{

    public class Checker

    {

        private int _currentLevel = 0;

        private int _maxLevel;

        private int _pageMaxLevel = 0;

        private List<string> _messages;

        private int[] _pagesMaxLevel;

        public int MaxLevel { get { return _maxLevel; } }

        public int[] PagesMaxLevel { get { return _pagesMaxLevel; } }

        public string[] Messages { get { return _messages.ToArray(); } }

        public int Exec(string pdfFileName)

        {

            _maxLevel = 0;

            _messages = new List<string>();

            PdfReader reader = new PdfReader(pdfFileName);

            _pagesMaxLevel = new int[reader.NumberOfPages];

            for (int p = 1; p <= reader.NumberOfPages; p++)

            {

                _pageMaxLevel = 0;

                _messages.Add("Page: " + p);

                PdfDictionary pageDict = reader.GetPageN(p);

                PdfObject res = pageDict.Get(PdfName.RESOURCES);

                if (res.IsIndirect())

                {

                    PRIndirectReference pirr = res as PRIndirectReference;

                    PdfDictionary newRes = pirr.Reader.GetPdfObject(pirr.Number) as PdfDictionary;

                    res = newRes;

                }

                if (res != null)

                    InternalDictCheck(res as PdfDictionary);

                else

                {

                    // Non ci sono risorse??? Possibile????

                }

                _pagesMaxLevel[p - 1] = _pageMaxLevel;

            }

            reader.Close();

            return _maxLevel;

        }

        private void InternalDictCheck(PdfDictionary dict)

        {

            _currentLevel++;

            _messages.Add("   CurrentLevel: " + _currentLevel);

            if (_currentLevel > _maxLevel)

                _maxLevel = _currentLevel;

            if (_currentLevel > _pageMaxLevel)

                _pageMaxLevel = _currentLevel;

            PdfDictionary xobjDict = dict.Get(PdfName.XOBJECT) as PdfDictionary;

            if (xobjDict != null)

            {

                foreach (PdfName obj2Name in xobjDict.Keys)

                {

                    _messages.Add("   obj2Name: " + obj2Name);

                    PRIndirectReference pirr = xobjDict.Get(obj2Name) as PRIndirectReference;

                    PdfDictionary dict2 = pirr.Reader.GetPdfObject(pirr.Number) as PdfDictionary;

                    PdfDictionary dict3 = dict2.Get(PdfName.RESOURCES) as PdfDictionary;

                    if (dict3 != null)

                        InternalDictCheck(dict3);

                }

            }

            _currentLevel--;

        }

    }

Apache PDFBox is a useful Java library for working with PDF documents. It allows you to create new PDF documents and extract data from existing documents.

However, the library doesn't provide an API for creating tables within PDF documents. So I wrote my own method which uses basic operations like drawLine to draw the table cells and drawString to fill in the content. The code is shown below. Note, that this code could be improved to handle longer strings of text properly by wrapping text within cells, which it doesn't do at present.

This post describes how I created a PDF document from a set of twitter statuses. I created a XSLT stylesheet transforming a twitter status as XML. This stylesheet is available on github at: https://github.com/.../twitter/status2fo.xsl. The stylesheet transforms the XML file generated by the twitter API (e.g.:http://api.twitter.com/1/statuses/show/44175380516585472.xml) to XSL-FO. This xsl-fo is then processed by Apache-FOP to generate a PDF.

The apache project PDFBox contains is an API for handling some PDF documents. In the current post I'll show how I've used the PDFBox API to insert and extract some XMP metadata into/from a PDFDocument.

Extracting metadata from a PDF document

Reading the metadat is as simple as:

InputStream in=new FileInputStream(pdfFile); PDFParser parser=new PDFParser(in); parser.parse(); PDMetadata metadata = parser.getPDDocument().getDocumentCatalog().getMetadata(); if(metadata!=null) { System.out.println(metadata.getInputStreamAsString()); }

Inserting metadata into a PDF document

The metadata to be inserted are stored in a XML file.

<?xml version="1.0" encoding="UTF-8"?> <x:xmpmeta xmlns:x="adobe:ns:meta/"> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:dc="http://purl.org/dc/elements/1.1/"> <rdf:Description rdf:about=" "> <dc:creator rdf:resource="mailto:[email protected]"/> <dc:title>Hello World</dc:title> <dc:date>2010-07-11</dc:date> </rdf:Description> <foaf:Person rdf:about="mailto:[email protected]"> <foaf:name>Pierre Lindenbaum</foaf:name> <foaf:depiction rdf:resource="http://a3.twimg.com/profile_images/51679789/photoIG_bigger.jpg"/> </foaf:Person> </rdf:RDF> </x:xmpmeta>

This XML file is loaded as a DOM object in memory:

DocumentBuilderFactory f= DocumentBuilderFactory.newInstance(); f.setExpandEntityReferences(true); f.setIgnoringComments(true); f.setIgnoringElementContentWhitespace(true); f.setValidating(false); f.setCoalescing(true); f.setNamespaceAware(true); DocumentBuilder builder=f.newDocumentBuilder(); xmpDoc= builder.parse(xmpIn);

The pdf source is opened and the DOM document is inserted as a metadata. The pdf is then saved:

InputStream in=new FileInputStream(pdfIn); PDFParser parser=new PDFParser(in); parser.parse(); document= parser.getPDDocument(); PDDocumentCatalog cat = document.getDocumentCatalog(); PDMetadata metadata = new PDMetadata(document); metadata.importXMPMetadata(new XMPMetadata(xmpDoc)); cat.setMetadata(metadata); document.save(pdfOut);

Source code: ExtractXMP.java

import java.io.*; import org.apache.pdfbox.pdfparser.*; import org.apache.pdfbox.pdmodel.*; import org.apache.pdfbox.pdmodel.common.*; import org.apache.jempbox.xmp.XMPMetadata; public class ExtractXMP { static private void extract(InputStream in) throws Exception { PDDocument document=null; try { PDFParser parser=new PDFParser(in); parser.parse(); document= parser.getPDDocument(); if(document.isEncrypted()) { System.err.println("Document is Encrypted!"); } PDDocumentCatalog cat = document.getDocumentCatalog(); PDMetadata metadata = cat.getMetadata(); if(metadata!=null) { System.out.println(metadata.getInputStreamAsString()); } } catch(Exception err) { throw err; } finally { if(document!=null) try { document.close();} catch(Throwable err2) {} } } static public void main(String args[]) { try { int optind=0; while(optind<args.length) { if(args[optind].equals("-h")) { System.err.println("Pierre Lindenbaum PhD. 2010"); System.err.println("-h this screen"); System.err.println("pdf1 pdf2 pdf3 ...."); return; } else if (args[optind].equals("--")) { ++optind; break; } else if (args[optind].startsWith("-")) { System.err.println("bad argument " + args[optind]); System.exit(-1); } else { break; } ++optind; } if(optind==args.length) { extract(System.in); } else { while(optind< args.length) { String filename=args[optind++]; InputStream in=new FileInputStream(filename); extract(in); in.close(); } } } catch(Throwable err) { err.printStackTrace(); } } }

Source code: InsertXMP.java

import java.io.*; import org.apache.pdfbox.pdfparser.*; import org.apache.pdfbox.pdmodel.*; import org.apache.pdfbox.pdmodel.common.*; import org.apache.jempbox.xmp.XMPMetadata; import javax.xml.parsers.DocumentBuilder; import javax.xml.parsers.DocumentBuilderFactory; import org.w3c.dom.Document; public class InsertXMP { static public void main(String args[]) { PDDocument document=null; InputStream in=null; try { String xmpIn=null; String pdfIn=null; String pdfOut=null; Document xmpDoc=null; int optind=0; while(optind<args.length) { if(args[optind].equals("-h")) { System.err.println("Pierre Lindenbaum PhD. 2010"); System.err.println("-h this screen"); System.err.println("-pdfin|-i <pdf-in>"); System.err.println("-xmpin|-x <xmp-in>"); System.err.println("-pdfout|-o <pdf-out>"); return; } else if(args[optind].equals("-xmpin") || args[optind].equals("-x")) { xmpIn= args[++optind]; } else if(args[optind].equals("-pdfin") || args[optind].equals("-i")) { pdfIn= args[++optind]; } else if(args[optind].equals("-pdfout") || args[optind].equals("-o")) { pdfOut= args[++optind]; } else if (args[optind].equals("--")) { ++optind; break; } else if (args[optind].startsWith("-")) { System.err.println("bad argument " + args[optind]); System.exit(-1); } else { break; } ++optind; } if(optind!=args.length) { System.err.println("Illegal number of arguments"); return; } if(pdfIn==null) { System.err.println("pdf-in missing"); return; } if(pdfOut==null) { System.err.println("pdf-out missing"); return; } if(pdfIn.equals(pdfOut)) { System.err.println("pdf-out is same as pdf-in"); return; } if(xmpIn==null) { System.err.println("XMP missing"); return; } else { DocumentBuilderFactory f= DocumentBuilderFactory.newInstance(); f.setExpandEntityReferences(true); f.setIgnoringComments(true); f.setIgnoringElementContentWhitespace(true); f.setValidating(false); f.setCoalescing(true); f.setNamespaceAware(true); DocumentBuilder builder=f.newDocumentBuilder(); xmpDoc= builder.parse(xmpIn); } in=new FileInputStream(pdfIn); PDFParser parser=new PDFParser(in); parser.parse(); document= parser.getPDDocument(); if(document.isEncrypted()) { System.err.println("Warning ! Document is Encrypted!"); } PDDocumentCatalog cat = document.getDocumentCatalog(); PDMetadata metadata = new PDMetadata(document); metadata.importXMPMetadata(new XMPMetadata(xmpDoc)); cat.setMetadata(metadata); document.save(pdfOut); } catch(Throwable err) { err.printStackTrace(); } finally { if(document!=null) try { document.close();} catch(Throwable err2) {} if(in!=null) try { in.close();} catch(Throwable err2) {} } } }

Example

The following Makefile downloads a pdf file, compiles both program , inserts and extracts the metadata:

CLASSPATH=pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. test: InsertXMP.class ExtractXMP.class article.pdf echo "Metadata in article" java -cp ${CLASSPATH} ExtractXMP article.pdf echo "Insert Metadata in article" java -cp ${CLASSPATH} InsertXMP -i article.pdf -o article_meta.pdf -x metadata.xmp echo "Metadata in new article" java -cp ${CLASSPATH} ExtractXMP article_meta.pdf InsertXMP.class:InsertXMP.java javac -cp ${CLASSPATH} InsertXMP.java ExtractXMP.class:ExtractXMP.java javac -cp ${CLASSPATH} ExtractXMP.java article.pdf: wget -O $@ "http://www.biomedcentral.com/content/pdf/1471-2156-10-16.pdf"

Output

javac -cp pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. InsertXMP.java javac -cp pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. ExtractXMP.java wget -O article.pdf "http://www.biomedcentral.com/content/pdf/1471-2156-10-16.pdf" --2010-07-11 13:15:10-- http://www.biomedcentral.com/content/pdf/1471-2156-10-16.pdf Resolving www.biomedcentral.com... 213.219.33.18 Connecting to www.biomedcentral.com|213.219.33.18|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1295524 (1.2M) [application/pdf] Saving to: `article.pdf' 100%[================================================================================>] 1,295,524 123K/s in 11s 2010-07-11 13:15:21 (113 KB/s) - `article.pdf' saved [1295524/1295524] Metadata in article java -cp pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. ExtractXMP article.pdf <?xpacket begin="" id="W5M0MpCehiHzreSzNTczkc9d"?> <x:xmpmeta xmlns:x="adobe:ns:meta/" x:xmptk="3.1-701"> <rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:Description rdf:about="" xmlns:pdf="http://ns.adobe.com/pdf/1.3/"> <pdf:Producer>Acrobat Distiller 7.0 (Windows)</pdf:Producer> </rdf:Description> <rdf:Description rdf:about="" xmlns:xap="http://ns.adobe.com/xap/1.0/"> <xap:CreateDate>2009-05-05T19:28:38Z</xap:CreateDate> <xap:CreatorTool>FrameMaker 7.1</xap:CreatorTool> <xap:ModifyDate>2009-05-06T02:18:59+05:30</xap:ModifyDate> <xap:MetadataDate>2009-05-06T02:18:59+05:30</xap:MetadataDate> </rdf:Description> <rdf:Description rdf:about="" xmlns:dc="http://purl.org/dc/elements/1.1/"> <dc:format>application/pdf</dc:format> <dc:title> <rdf:Alt> <rdf:li xml:lang="x-default">1471-2156-10-16.fm</rdf:li> </rdf:Alt> </dc:title> <dc:creator> <rdf:Seq> <rdf:li>Ezhilan</rdf:li> </rdf:Seq> </dc:creator> </rdf:Description> <rdf:Description rdf:about="" xmlns:xapMM="http://ns.adobe.com/xap/1.0/mm/"> <xapMM:DocumentID>uuid:d1d0f8d9-8321-4e4b-828b-d31b75daba0f</xapMM:DocumentID> <xapMM:InstanceID>uuid:39d7db98-d873-4b33-be85-87319547e81c</xapMM:InstanceID> </rdf:Description> </rdf:RDF> </x:xmpmeta> <?xpacket end="w"?> Insert Metadata in article java -cp pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. InsertXMP -i article.pdf -o article_meta.pdf -x meta.xmp Metadata in new article java -cp pdfbox-app-1.2.1.jar:pdfbox-app-1.2.1.jar:. ExtractXMP article_meta.pdf <x:xmpmeta xmlns:x="adobe:ns:meta/"> <rdf:RDF xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:foaf="http://xmlns.com/foaf/0.1/" xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"> <rdf:Description rdf:about=""> <dc:creator rdf:resource="mailto:[email protected]"/> <dc:title>Hello World</dc:title> <dc:date>2010-07-11</dc:date> </rdf:Description> <foaf:Person rdf:about="mailto:[email protected]"> <foaf:name>Pierre Lindenbaum</foaf:name> <foaf:depiction rdf:resource="http://profile.ak.facebook.com/profile5/1306/97/s501154465_2583.jpg"/> </foaf:Person> </rdf:RDF> </x:xmpmeta>

How to do text extraction from pdf files using PdfBox on Google App Engine (Warning: I used an old version of PdfBox: 0.7.3.) PdfBox is a very popular Java library for creating and managing pdf files. It's also able to extract text from existing pdf files. Pdfbox is published as a jar file. I'd like to use it on Google App Engine (java version) for text extraction from particular area of the page of pdf files. PdfBox allows that. The class to use is PDFTextStripperByArea. I tried it but GAE blocked me: PDFTextStripperByArea uses not allowed JRE classes. In particular jawa.awt.Rectangle and Rectangle2D. GAE applies a "white list" approach: only a subset of the standard JRE classes is allowed to run on GAE. 99% of Java.awt.* is blocked. http://code.google.com/appengine/docs/java/jrewhitelist.html There is also another problem. During text extraction PdfBox uses a temp file. By default it's created on the file-system. GAE also blocks the access to the file-system. My solution was:

use my own Rectangle instead of java.awt.Rectangle

use a "in memory" temp file

PaaS is a type of cloud service in which the provider delivers not only on-demand hardware and operating-system services, but also application platforms and solution stacks. PaaS services automate most of the IT management aspects associated with application deployment, including resource allocation, staging and testing, load balancing, database access, and access to platform libraries. A key feature of PaaS is multitenant architecture: multiple unrelated applications can run on the same hardware and software infrastructure, resulting in cost savings and more-efficient use of computing resources. Developers can focus on the application itself, as opposed to deployment and IT issues.

Java developers are well positioned to learn and take advantage of the PaaS development model. After all, the PaaS concept has deep roots in the early days of server-side Java. Back then, IT organizations were sold on the vision of using application servers as "containers" and then dropping in application-archive files to run in a shared-resource environment (see the PaaS precursor sidebar). That vision is remarkably similar to the PaaS services we see today.

But the early PaaS vision of Java enterprise applications didn't pan out. Java application servers never became stable enough to deploy and undeploy multiple unrelated applications at will. The archive structure ended up adding overhead to the Java application-development cycle: Whereas PHP and Ruby on Rails developers can change a line of code and reload the browser to see the difference, Java developers had to recompile, repackage, and redeploy their applications, and often even restart the application server.

As it turns out, the PaaS vision is only realized with the emergence of a new generation of virtualization technologies that are far more advanced than the JVM. Pioneered by Google App Engine, the new generation of Java PaaS services fulfills the old promise of Java EE. And they provide pay-as-you-need IT infrastructures that grow with your demand without requiring you to invest up front in expensive hardware and system-administration capacities.

In this article, I'll examine three leading Java public PaaS offerings to compare their approaches, strengths, and weaknesses. All three provide the same basic set of features, including:

Uploading and deploying application WARs

Versioning deployed applications

Testing and staging environments

Online access to log files

Automated monitoring and usage reports

Beyond those common features, however, are some important differences. Drawing from my own experience working with these emerging technologies, I'll offer a framework for comparing them and discuss potential workarounds to help you avoid problems when you use them.

Google App Engine

Google App Engine (GAE) is the first widely adopted Java PaaS platform. (The Java version is sometimes called GAE/J to distinguish it from GAE's Python-based PaaS offering.) It is also probably the "purest" PaaS offering on the market — in the sense that it almost completely abstracts away the underlying infrastructure from developers.

Java, but not quite Java

GAE has supported the Java platform as a development and deployment environment since 2009. However, GAE's Java support is limited and not standards-compliant. Because of numerous restrictions it imposes on its applications — many of them for good reasons to maintain scalability — GAE doesn't support certain Java platform APIs: most noticeably, file write I/O (because GAE doesn't provide a file system accessible to applications) and many of the network I/O APIs (because GAE imposes severe limitations on network operations originated from the applications). See Resources for a complete list of "white listed" Java platform APIs supported by GAE.

By supporting its own limited network I/O API, GAE restricts an application's capacity to connect to other services. GAE nominally allows applications to make outbound connections to other servers. But in an effort to keep the number of threads in the system under control, GAE forces any application-initiated connection to close after 5 to 10 seconds. That makes GAE an unreliable platform for mashup-type applications. This is a major limitation of GAE for the increasing number of applications that make use of third-party web services APIs.

Furthermore, those API limitations impose challenges when you need to use existing application frameworks or move existing applications to GAE. After years of evolution, enterprise Java development is heavily dependent on frameworks. Although some popular frameworks, such as Spring and Struts, work out of the box on GAE, many others either do not work or require patches to their source code. Manually hacking framework source code to make it run on GAE is never a good idea, because you are essentially creating a fork that breaks upstream compatibility, and it could introduce hard-to-debug bugs into the framework. A good example is the JavaServer Faces (JSF) web framework: it requires source-code level hacking to run in the GAE environment, and even then many UI libraries on top of JSF are incompatible with GAE. (See Resources for a list of GAE-supported Java frameworks.)

Likewise, large enterprise applications that are already developed are likely to use APIs that GAE prohibits. Migrating those applications to GAE might be costly, because you not only need to identify the issues and create workarounds, but also do quality assurance for the entire application all over again.

By not supporting part of the Java platform API, GAE breaks Java's promise of "write once, run anywhere." It's not a deal-breaker for many people, but it is something potential users need to be aware of.

Scalability and performance

GAE promises and delivers scalability but not necessarily raw performance. Raw performance for web applications is measured by response time to a web request. Scalability refers to the platform's ability to maintain a consistent response time regardless of how many users are accessing the system. For instance, a scalable system with a response of 3 seconds for 100 concurrent users should have a 3-second response time for 1 million concurrent users.

GAE provides excellent scalability as measured by a consistent response time. But its raw performance is often slow. In my own anecdotal experience, GAE often takes 1 to 3 seconds to respond to database-related requests.

That characteristic has obvious implications for application developers. For web applications that are idle most of the time (namely, most small web applications), deploying on the GAE infrastructure would not yield performance benefits over even a low-end virtual private server. The real performance benefit comes when you need to scale the application massively well beyond the capacity of low-end server hardware.

Another performance issue for low-traffic websites is that GAE swaps inactive JVMs out of memory to optimize for high-traffic web applications on the system. If your JVM is swapped out of memory, GAE must spend additional time to start your entire application the next time a request comes in. For low-traffic web applications, this could lead to slow performance (more than 5 seconds waiting time for a first request). GAE offers an option for developers to pay and keep the inactive JVM in memory for more-consistent performance. One tip: Set up a cron job inside GAE to load your own website every 2 to 3 minutes to keep the JVM active.

Benefits and limitations of BigTable

A key innovation of GAE is the use of a truly scalable data store: Google BigTable. Most web applications use relational databases as data back ends. But relational databases are notoriously difficult to scale. To solve this problem, researchers at Google developed an alternative data-storage solution called BigTable, one of the data-storage solutions in the world of NoSQL databases.

As in a relational database, data in BigTable can be organized into tables with rows and columns, and each row has a unique indexed ID. Unlike relational databases, BigTable tables do not have fixed schema and are typically denormalized. Each row in a table could have different columns. The best practice is to have many columns in a row, as opposed to linking different rows across different tables via key columns. That has big implications for the design of the data models. Instead of designing a normalized relational model, application developers are encouraged to put redundant information into each row for easier retrieval. Think of the access log of a web server where the IP address and browser agent are repeated in every row, taking up space but simplifying bulk processing.

The benefit of BigTable is scalability. Google engineers claim that the response time of data queries in BigTable is only determined by the size of the result dataset. You get the same performance no matter if the query is against a 1,000-row table or a 10-million-row table, as long as the result is limited to 1,000 rows. For its part, GAE limits the returned dataset of each query to 1,000 rows.

Adjusting to the NoSQL paradigm, although it could be challenging for developers from an SQL background, is an important skill to have as more and more IT organizations are facing the Big Data challenge. I have found that GAE is one of the best and easiest places for Java developers to get started learning NoSQL.

However, although BigTable is key to GAE's massive scalability, its current implementation leaves a lot to be desired for Java developers. Specific shortcomings of BigTable (and some potential workarounds) include:

Weak support for data queries: Queries written in Google Query Language (GQL) are used to retrieve data from BigTable. GAE requires that all the data columns involved in a query be indexed, and the index can't contain BLOB or text columns. That's fine, except that GAE allows only 100 indexes per table. That's probably sufficient for a standard SQL database, but denormalized NoSQL databases like BigTable could potentially have thousands of columns, so 100 indexes could be limiting for many applications. Making matters worse, GAE provides no easy way to delete indexes that are no longer in use.

Deciding which index to create is a significant burden for GAE developers. If a query uses a combination of columns that are not indexed, GAE will only throw an exception at run time when the query is executed. Although the SDK provides tools for automatically generating index-configuration files as you test the application on your local computer, you could still miss indexes if you do not manually test all the execution paths exhaustively. Merging the autogenerated indexes into an already deployed application is also a potentially error-prone process with no indication of error until web application users hit the misconfigured indexes.

Finally, it is somewhat shocking — considering that BigTable is a Google product — that it doesn't support free text search in the database. You could embed a search-engine implementation, such as Apache Lucene, into your application to index and search text columns (see Resources). But that's a big hassle for smaller websites for which standard SQL LIKE statements are sufficient for simple text search.

Difficulty importing and exporting data: Another major issue with BigTable is the inability to import and export data. Because there's no standard API for directly accessing BigTable, you must write data-import and data-export logic into servlets inside your own application, and use your own web interface to import or export data.

Because GAE terminates any web-request thread after 30 seconds, it's impossible to upload a large set of data into BigTable via a persistent connection. A common workaround is to break the data import into many pieces, with each piece requiring less than 30 seconds to upload and process. Then, you can use an automated HTTP driver, such as JMeter or Grinder, to run those tasks one by one until all data is imported. Needless to say, this is a tedious process.

Exporting data from BigTable is even more problematic. Because the API limits each data query to 1,000 results, the export data must be managed in even smaller chunks than the 30-second processing timeout constraint allows.

Recognizing the limitations of BigTable for most developers, GAE provides access to hosted MySQL services via its paid business offerings.

Integration with other services

GAE provides excellent integration with other Google services. Notably, the application can integrate with Google Accounts so that users can log into your application using a Google username and password. That could potentially save you a lot of time, given that building a user-management system is duplicate work every website has to do. However, the downsides are that not all users have Google accounts, and that tying your website to Google Accounts would make it hard to move to another PaaS provider later.

GAE applications can also use a simple API to send email messages via GMail servers. Compared with unsecured SMTP servers, GMail servers are much less likely to be blocked by the recipient ISP.

If you host your domain on Google Apps, you can also configure the application to be accessed via any subdomain under your control by linking your Google Apps account with your GAE account. For instance, if mydomain.com is hosted by Google Apps, you can make your application accessible from www.mydomain.com as opposed to mydomain.appspot.com.

Verdict

Overall, GAE provides a well-designed and scalable PaaS. Its generous free quota for small websites is also appealing. However, the lack of support for the complete Java platform is a potential deal breaker, and some of the components in GAE still feel experimental rather than production-ready.

Amazon Elastic Beanstalk

Amazon Elastic Beanstalk, a relatively new offering from Amazon Web Services, provides a managed Apache Tomcat runtime environment based on the Amazon Elastic Computing Cloud (EC2) infrastructure. EC2 is an Infrastructure-as-a-Service (IaaS) offering, so it provides much more flexibility than GAE. But as a trade-off, it also requires more developer effort to manage and scale the applications.

Pure Java Tomcat

The Beanstalk environment supports a full Tomcat server running on an EC2 virtual server. It is a pure Java environment with access to the underlying file system. Because of Tomcat's popularity, almost all enterprise Java frameworks support Tomcat deployment. Those frameworks can be started or bootstrapped from a Tomcat WAR file, offering you a wide variety of choices of frameworks and libraries.

The plain Tomcat runtime has no limitation on threading and file or network I/O. A network I/O thread can stay open as long as it is needed. You are only limited by the capacity of the underlying virtual machine.

Scaling, at a price

Beanstalk scales your application by automatically starting new EC2 instances and deploying your WAR file to the new instance. All your Beanstalk EC2 instances are running behind a load balancer. You can use a web-based management console to monitor the resources available on each EC2 instance and set up rules for automatically starting new server instances behind the load balancer when the existing server load exceeds preset limits.

A common issue in a load-balanced web cluster is how to handle HTTP sessions. Each Tomcat server node creates and manages session objects for its clients. If web requests are load-balanced across multiple server nodes, you need to make sure that the server node serving the request has the correct session object. A simplistic way to archive this is to enable "sticky session" in the load balancer, requiring the load balancer to remember the session cookies maintained by each server behind it, and forward requests to the right server based on the incoming cookies. The "sticky session" can be turned on in the Beanstalk load balancer administration console. More efficient and fail-safe solutions include setting up shared memory across the server nodes or simply saving the session objects into a central database. Those options allow the load balancer to forward requests to a random or the least-busy server node, because every server node has the same session-state information. But all those options require effort from the application developer. Unlike GAE, which automatically saves session data into BigTable, Beanstalk requires you to do all the work.

Perhaps one of Beanstalk's biggest drawbacks is its price, especially for small websites that can get free hosting elsewhere. While Amazon EC2 has a "one year free" program for new signups, Beanstalk's standard pricing runs close to $40 a month even for a single-node setup. That's a cheap price for a cluster-ready infrastructure that can scale out automatically in a matter of minutes when needed, but it's expensive compared with the likes of GAE if your application is mostly idle with an occasional surge of traffic.

Flexible database choices

One of the strengths of the Elastic Beanstalk platform is flexibility in choosing database technologies. It offers several options:

Relational databases: Through Amazon's own Relational Database Service (RDS), you can deploy a variety of relational databases. Those database servers are managed and monitored by Amazon, and it's easy to import data into and export it from them. Inside your application, all you need to do is to point your data sources to your RDS server. But be aware that each RDS instance is another dedicated server instance running your database — and a database instance is 30 percent more expensive than a comparable EC2 instance. The cost could add up, and many applications do not need a dedicated database server.

NoSQL: One issue with the RDS server is that it is a relational database that is hard to scale. If you prefer a NoSQL approach similar to Google BigTable, it's available too with Amazon SimpleDB. SimpleDB's Java API lets your application easily access the data.

Your own database server: Because EC2 provides access to raw virtual servers, you can set up your own databases or NoSQL data sources (such as Apache Cassandra) on a separate EC2 instance and just point the Beanstalk application to your own database server.

The flexibility in database choices, especially the ability to use Amazon managed relational databases, is likely to appeal to enterprise developers.

Integration with other services

In addition to Amazon RDS and SimpleDB, Beanstalk servers have access to other Amazon services such as Simple Queue Service, S3 Storage, Simple Email Service (SES), and payment APIs. SES is especially interesting and offers a good comparison point with the GMail API in GAE.

SES has a simple API, and it allows you to use Amazon's SMTP server to send out email messages. The benefit to using Amazon SMTP servers, as opposed to setting up an unsecured SMTP server on your own EC2 instance, is that Amazon servers are less likely to be blocked by major ISPs' spam filters. To this end, SES provides a rich set of tools to control the ramp-up of email volumes and to receive feedback from ISP spam filters. All those features are made available to your Beanstalk application so that you can monitor your campaigns and optimize your email content for more efficient delivery.

Verdict

Overall, Amazon Elastic Beanstalk greatly simplifies the deployment and scaling of Tomcat applications. Yet, it still provides the flexibility of the underlying EC2 infrastructure, which makes it ideally suited for enterprise applications. The cost, however, is high for low-traffic websites or hobbyist developers.

CloudBees RUN@Cloud

CloudBees is a new entrant to the Java PaaS scene. It may be a startup, but the people behind it are enterprise Java veterans. (It was started by JBoss ex-CTO Sacha Labourey, and has employed open source Java heavyweights Adrian Brock of JBoss fame and Kohsuke Kawaguchi of Hudson fame.) Its PaaS technology was acquired from Stax Networks, which has been providing hosted Java application services to enterprise customers for more than 10 years. The CloudBees RUN@Cloud service is based on the robust Stax platform, and it is available to individual developers via a self-service web portal.

In comparison with the big players, RUN@Cloud aims to find the right balance between managed scalability (as in GAE) and flexibility (as in Amazon's PaaS services) while adding its own twist of end-to-end development life-cycle support via the platform.

A robust Java runtime

The RUN@Cloud service is currently based on the EC2 infrastructure, and it can be viewed as a more automated version of Beanstalk + RDS. Like Beanstalk, RUN@Cloud also offers a dedicated Tomcat instance running on an EC2 virtual server for each web application. It provides a pure Java environment with no artificial limitation on file system access, network I/O, and threading.

One of RUN@Cloud's strengths as a small independent company is that it doesn't need to be tied with Amazon. It plans to offer other infrastructure providers to supplement EC2 in the near future.

Free scalable infrastructure

Also similar to Beanstalk, RUN@Cloud provides a scalable infrastructure with load balancer and server instances to be started on demand to meet traffic surges. But RUN@Cloud provides more automation than Beanstalk. For instance, instead of using "sticky sessions," RUN@Cloud has already configured its Tomcat servers to save sessions to databases under its management. This managed-session object database is transparent to developers — much like GAE.

Because RUN@Cloud can use a shared load balancer to manage multiple Tomcat servers running on a single EC2 instance, it does not require one EC2 instance per Tomcat instance. Hence it can run low-traffic websites at much lower cost than Beanstalk. In fact, RUN@Cloud has a free usage tier that is great for low-traffic applications or hobbyist developers and students.

However, also like GAE, RUN@Cloud can swap your JVM out of memory if your application is inactive for too long, to conserve resources. That could cause slow response to the first request as the application "warms up."

Hosted MySQL relational databases

The RUN@Cloud service natively supports a managed MySQL service alongside the Tomcat service. You can create and manage databases through a web-based administration console. And you can connect to the database server directly via a MySQL client in order to manage your data.

Unlike Amazon RDS, the RUN@Cloud service deploys a shared database server across multiple applications. Each application can have its own database but not necessarily a dedicated server. The PaaS platform automatically deploys the database to maximize the utilization of a pool of database servers. Compared with RDS, the shared database server would yield potentially more-efficient use of virtual servers, and hence lower costs.

Integration with other services

RUN@Cloud provides access to platform APIs and services supported by its underlying infrastructure providers. Specifically for RUN@Cloud applications deployed on Amazon EC2, those applications have full access to all Amazon web service APIs — such as S3, SQS, and SES — from inside your application.

But where RUN@Cloud really shines is its tight integration with DEV@Cloud, a cloud-based, Continuous Integration platform. DEV@Cloud provides source-code, version-control systems (Subversion and GIT); a build repository (Apache Maven); and a build server (jenkins, formerly called Hudson). It allows you to run automated building and testing of your applications in the cloud rather than on your own computer. This type of centralized build system is widely adopted by agile software teams to make sure that the source code in the repository is always tested and in a releasable state.

By integrating RUN@Cloud with DEV@Cloud, CloudBees provides a compelling set of PaaS services that can manage the entire development, testing, and deployment cycle of enterprise Java web applications. You just need to edit source code on your own computer, and everything else can be delegated to an automated system in the cloud with minimal IT overhead.

Verdict

CloudBees RUN@Cloud is a low-cost (and even free) alternative to Amazon Elastic Beanstalk and RDS. Its integration with continuous-build systems make it appealing to agile software development teams that wish to automate all the IT functions in the development process.

Conclusions

After years of disappointment, Java PaaS services have finally reached prime time. The three services reviewed and compared in this article each has its unique approach and, as a result, unique strengths and weaknesses.

If you are developing a new application and can live with GAE's constraints, GAE is an excellent and free choice. RUN@Cloud and Elastic Beanstalk are interexchangable runtimes at the application level. Standard Java EE applications can run on either platform unmodified. RUN@Cloud is cheaper to get started with and easier to configure, and it provides excellent support for continuously integrated development processes. I suggest starting with RUN@Cloud for free, knowing that you can easily move to Elastic Beanstalk if you are unhappy with CloudBees' services.

Resources

Learn

"Gartner Says 2011 Will Be the Year of Platform as a Service": Excitement around PaaS continues to grow, as evidenced by this Gartner press release.

"Consider PaaS in Your Cloud Strategy": Gartner analyst Yefim Natis makes the case for PaaS in this presentation.

Google App Engine for Java: GAE for Java is one of the first Java PaaS offerings.

Amazon Web Services: Amazon Web Services offers a set of interlinked services including Elastic Beanstalk, Elastic Compute Cloud, Relational Database Service, SimpleDB, and Simple Email Service.

CloudBees: The CloudBees DEV@Cloud and RUN@Cloud services both have a free layer for small websites.

Java development 2.0: Andrew Glover's series of articles on developerWorks covers how to get started with Google App Engine, BigTable, and Amazon Elastic Beanstalk.

"Google App Engine for Java": Check out Richard Hightower's three-part article on how to build Java applications for GAE.

The JRE Class White List: Google's official list of Java platform APIs supported in GAE.

WillItPlayInJava: View the status of GAE's support for popular frameworks.

Browse the technology bookstore for books on these and other technical topics.

developerWorks Java technology: Find hundreds of articles about every aspect of Java programming.

Get products and technologies

Apache Lucene: You can build your own text-search engine using Lucene on top of BigTable to enable search queries in your GAE applications.

Discuss

Get involved in the developerWorks community.

About the author