Parsing PDF Digital Form Values
Today we'll look into how to parse PDF Digital Forms values with PDFHummus. Code examples are built with the NodeJS flavor, HummusJS, However the same principles (and module names, mostly) go for C++ usages.
What I am looking for is to be able to get a PDF file that includes a form that somebody filled, and to be able to get the values that they filled. This data be available in a convenient representation and become actionable.
For the example I built a module that parses form values which you are welcome to reuse.
you can find it here. pdf-digital-form.js is where it resides. main.js holds a usage example.
To use it, create an instance of the class it provides and lend it a PDFReader instance, like this:
var hummus = require('hummus'), PDFDigitalForm = require('./pdf-digital-form') var pdfParser = hummus.createReader('the-pdf-file-path'); var digitalForm = new PDFDigitalForm(pdfParser);
Then you can call its hasForm() method, which will let you know if it has a form, and read its fields member to get a hierarchical view of the form fields value data (as we'll see later fields may be organized in a tree form to allow some semantic or syntactic grouping).
You can also get a simplified flattened view by calling its createSimpleKeyValue() method which will return a JS Object where its keys are the full field names, and values are their matching values.
if(digitalForm.hasForm()) { console.log(digitalForm.fields); console.log(digitalForm.createSimpleKeyValue()); }
Let's look into a little bit of theory about PDF Forms, so we understand how to parse it. Later we'll see how to build PDFDigitalForm.
In A PDF file, A PDF Form has two typs of representations.
One is logical, in which fields are described by their types (text, checkbox etc.) and values. It is found in the AcroForm object of the PDF Catalog. The AcroForm is a dictionary which holds a Fields array. The fields in the array may be leaf fields which represent actual fields like a specific radio button or checkbox. Otherwise, they may be semantic sub-groups of fields represented by a single field. Each such sub-group has its own Fields array to describe its fields. This goes on till one gets to leaf fields.
Another representation exists for fields, and it is describing the appearance of the fields. Each leaf field as a matching Widget Annotation. This annotation is linked from the leaf fields object, but also from the page annotations list, in which this annotation is to be posited. The annotation is placed on the page in the coordinates where you want the form fields to appear.
Since in this post we're looking at fields values, we'll be naturally more interested in the first representation.
Any form fields may be of one of the following types:
Push Button : an action button which executes something. naturally, it holds no values. normally used for submitting the form.
Radio Button : A group of one or more buttons forming a set, where only one of them can be selected. The selected button represents the radio button value.
Check box : A toggle button which may be on or off. Its value is a Boolean matching whether its on or off.
Text or Rich Text : text field which value is free text input by the user. Rich Text fields allow HTML kind of input which can represent rich text as oppose to the plain text fields represented by Text fields.
Choice : Selection box. Optionally with a connected text field ("Combo"). May allow for multiple choices.
Signature : A signature field tying to PDF Signature mechanism.
Signature fields values are highly related to how signatures are implemented in PDF, and I don't think that is interesting for an automatic retrieval of form values kind of implementations. Push Buttons are naturally not very interesting either. So we will only look into how to parse other fields types.
As noted before fields are organized in a tree structure (though normally there would just be 1 level) to provide semantic representation of fields sub groups. This means that The root Fields array will contain Field dictionaries that may have their own Kids entry which is a fields array.
The tree structure defines inheritance for some properties, such as a field type. This means that at a certain level a field type will be defined, but this does not mean that the field is of this type - it may simply mean that it is a single semantic parent to multiple kids that have the same type. Only leaf fields have an actual real type.
For some leaf fields the Kids entry will be holding a pointer to their matching Widget Annotation which represents how the fields appear. In case of a radio button there will be multiple such annotations, each representing a single radio button. At other times the Widget Annotation will be combined with the leaf field dictionary itself to save the extra Kids structure. This is made possible because the keys defining a widget annotation object are exclusive to those that define a form field. However, this makes things a little confusing, As you cant tell if such a widget parent is a leaf field, or simply a parent of a leaf field which got combined with its annotation. We will be using some thumb rule to discern between the cases.
You can read more about PDF Forms in section 8.6 of the PDF Reference. I keep a copy of it here.
Parsing form values with Hummus
Now let's look into how to parse the values of a form with HummusJS. Again, anything specified here can also be done with the C++ equivalent, PDF-Writer.
Getting the acroform object
The AcroForm dictionary is linked from the PDF Catelog object and is what holds the form data. If it's not there - the PDF has no form.
We fetch it with the following code:
var catalogDict = pdfParser.queryDictionaryObject( pdfParser.getTrailer(), 'Root') .toPDFDictionary(), var acroformDict = catalogDict.exists('AcroForm') ? pdfParser.queryDictionaryObject(catalogDict,'AcroForm'): null; return acroformDict && acroformDict.toPDFDictionary();
The catalog is retrieved by getting the Root entry of the PDF trailer. It is checked for the existance of an AcroForm entry, and if exists fetches it. We're using pdfParser.queryDictionaryObject to skip indirect references in case the sub-dictionaries we're looking for are not directly embedded in their parent dicts. It's a generally good approach, unless you are Positive that the entry is embedded.
The top level fields are retrieved by querying the Fields entry form the acroform dict, like this:
var fieldsArray = acroformDict.exists('Fields') ? pdfParser.queryDictionaryObject( acroformDict, 'Fields').toPDFArray() : null;
This gives us a PDF array which we can iterate on and parse its kids. Something similar will happen with each field, if it has a Kids entry. The kids array is simply another fields array which can be parsed on in a similar manner to the top level fields array. Saying you got a fieldDict, getting is kids is done like this:
var fieldsArray = fieldDict.exists('Kids') ? pdfParser.queryDictionaryObject( fieldDict, 'Kids').toPDFArray() : null;
All this forms a recursion, which one stopping condition is the nonexistence of a "Kids" disctionary in the current stage fieldsDict. Note that in leaf levels the kids array is only holding annotations and not actual fields. We will discuss how to mitigate this later.
Two things of note in this recursion structure.
One - there are inheritable attributes. For our purpose the interesting ones are FT which denotes a field type and Ff which is a bit-field which can further split the field type. For instance All buttons have the same FT, and you discern push button from checkbox or radio button using the Ff. As such, we will be passing an inherited-properties parameter to the recursion. Initially it will be empty. For each stage, prior to parsing Kids reading the current fields and appending to the existing list of inherited properties looks a bit like this:
var localEnv = {} if(fieldDictionary.exists('FT')) localEnv['FT'] = fieldDictionary.queryObject('FT').toString(); if(fieldDictionary.exists('Ff')) localEnv['Ff'] = fieldDictionary.queryObject('Ff').toNumber(); var newEnv = _.extend({},inheritedProperties,localEnv);
inheritedProperties is the inherited object of properties. We're using lodash extend method, which simply builds the accumulated properties of what.s inherited and what's new. newEnv will become the inheritedProperties of the next level.
Another thing to pass via the recursion is the field name. We will want to build a "full name" in the return data for each field. This will be a simple concatenation of partial fields names. For example, if a field is called "source" and it has sub-children called "direct", "refer" or "social" then their full name would be "source.direct", "source.refer" and "source.social". For this purpose, in each recursion step we will be passing the concatenation of all current names, so that the next level full name, for each field, will be simply achieved be concatenating its own partial name.
Parsing the field dictionary
Iterating the fields array we get fields objects, which are dictionaries.
Parsing a single field object we will want to get the following: - its name. Do this by grabbing its T entry. - its value. do this by grabbing its V entry (for rich text it will be its RV entry) - its kids. via Kids entry - its type. via FT entry - its bit field. via Ff entry Note that some of these values are inheritable, which means that if they don't exist we take them from the inheritedProperties input value.
here is code to get all of them:
var fieldName = fieldDictionary.exists('T') ? fieldDictionary.queryObject('T').toPDFLiteralString().toText(): undefined, value = fieldDictionary.exists('V') ? fieldDictionary.queryObject('V'): undefined, kids = fieldDictionary.exists('Kids') ? pdfParser.queryDictionaryObject(fieldDictionary,'Kids').toPDFArray(): undefined, type = fieldDictionary.exists('FT') ? fieldDictionary.queryObject('FT').toString(): undefined, flags = fieldDictionary.exists('Ff') ? fieldDictionary.queryObject('Ff').toNumber(): undefined; if(flags === undefined) flags = inheritedProperties('Ff'); if(type === undefined) type = inheritedProperties('FT');
Note that we're using toPDFLiteralString().toText() for T. This is because its value is an encoded text string. In PDF this is a string that might be Unicode. Hummus has a PDFTextString class to handle these cases and provide the Unicode text string as a plain Javascript string. The PDFLiteralString object which is the type of the T entry has a toText method which goes through PDFTextString usage provide the required string directly.
toString used for FT and toNumber used for Ff are convenience methods avoiding the necessary to go through a toPDFXXXX conversion and get their string/number values directly - where you know that this entry should represent a string/number.
queryObject is used most of the time because the values are expected to be directly embedded in the dictionary. With Kids there might be a surprise so i'm using pdfParser.queryDictionaryObject.
Parsing the value of a field
The method of getting the value of a field depends on its type. Other than rich text we are always looking at the V entry, however every time we interpret it a little different.
For texts their value may be either a literal string or a stream. For literal strings grab the value directly and turn it to text. For stream, you need to read the stream data, put it in a PDFTextString and use its toString method to convert to text. In any case you end up with text.
For regular text field you take it from the V entry. For rich texts the V entry will hold the plain text representation of the input text, but RV will have the full rich text representation (in XHTML).
To know whether this is a rich text or regular text field consult the Ft value. It`s 26 bit should be 1 for this to be a rich text value. check like this:
Depending on whether you read the text from V or RV the reading code should look like this:
var valueField = pdfParser.queryDictionaryObject(fieldDictionary,fieldName); if(valueField.getType() == hummus.ePDFObjectLiteralString) { return valueField.toPDFLiteralString().toText(); } else if(valueField.getType() == hummus.ePDFObjectStream) { var bytes = []; var readStream = pdfReader.startReadingFromStream(valueField.toPDFStream()); while(readStream.notEnded()) { var readData = readStream.read(1); bytes.push(readData[0]); } return new PDFTextString(bytes).toString(); } else { return undefined; }
For streams we are using the built in startReadingFromStream to create a simple byte reader. Accumulating the bytes we are in the end submitting them to a new instance of PDFTextString to get the final text.
The select box field can provide either a single value or multiple values. In the first case V would be a single string value. In the latter case it would be an array of such values. end result code looks like this:
var valueField = pdfParser.queryDictionaryObject(fieldDictionary,"V"); if(valueField.getType() == hummus.ePDFObjectLiteralString) { return valueField.toPDFLiteralString().toText(); } else if(valueField.getType == hummus.ePDFObjectArray) { var arrayOfStrings = valueField.toPDFArray().toJSArray(); return _.map(arrayOfStrings,function(aString){ return aString.toPDFLiteralString().toText(); }); } else { return undefined; }
The usage of toJSArray converting the PDF array to a plain Javascript array, allows using the lodash map method creating a new array of this array with the strings as plain text. Win.
Parsing checkbox is easy, once you get the role of the Widget Annotation for checkbox. The annotation has two appearance modes ("Streams") - one for off and one for on. They are referred to by name. The off appearance has to be called "Off", and the other one...something else. The value of checkbox is simply the name of the appearance that should appear now. If it is unchecked, it should be 'Off'. If it is checked it should be that other name. So just check for Off or empty and you can tell if the checkbox is off. Like this:
var value = fieldDictionary.queryObject('V').toString(); if(value === 'Off' || value === '') { return false; } else return true;
Radio buttons parsing is slightly similar to checkboxes only there's a little bit more here. Radio buttons, as oppose to checkboxes, don't simply have an "on" or "off". They are either all "off" or that one of them is "on".
The value of the Radio button in V will be the appearance name for the "on" appearance of the radio button that should be on.
So. If the value of the radio button is "Off" none of them is selected. If it is not "Off" then we should look in its Kids array for the button that has this appearance name. Its index would be the result. Like this:
var value = fieldDictionary.queryObject('V').toString(); if(value === 'Off' || value === '') { return null; } else { var result = true; if(fieldDictionary.exists('Kids')) { var kidsArray = pdfParser.queryDictionaryObject( fieldDictionary, 'Kids').toPDFArray(); for(var i=0;i<kidsarray.getlength var widgetdictionary="pdfParser.queryArrayObject(kidsArray,i).toPDFDictionary();" apdictionary="pdfParser.queryDictionaryObject(" nappearances="pdfParser.queryDictionaryObject(" if found result="i;" save the selected index as value break return each kid is looked for its key which holds appearances and normal mode apperances. then looking to see it has this appearance name. so current result. note that sometimes there no kids object radio button actually used implement a checkbox. i don know why people do this...but true case. all we got values now summary bye ok. parsed values. still some bits piecese connecting codes discussed you are welcome look into module in examples complete also grab want your own usage. good luck xoxo gal.></kidsarray.getlength>
