#prerender.io

In today's fiercely competitive digital landscape, ensuring your website ranks well on search engines is non-negotiable. However, for websites primarily constructed using JavaScript frameworks such as React or Angular, achieving optimal search engine visibility can be challenging. This is because search engine crawlers often struggle to interpret and index dynamic content generated by JavaScript. Enter Prerender.io, with Redis for SEO, offering a robust solution to this dilemma. This guide will walk you through the process of installing Prerender.io, configuring it with Redis cache for improved performance, and seamlessly managing it with PM2 for efficient operation.

Prerequisites:

- Node.js and npm: These are essential for running JavaScript applications like Prerender.io. Install them using your system's package manager. - Redis: As a widely used in-memory data store, Redis serves as the caching layer for Prerender.io, significantly enhancing performance. Refer to the official Redis website for installation instructions. - Google Chrome: Prerender.io utilizes a headless Chrome instance for rendering web pages. Depending on your Linux distribution: - For Ubuntu/Debian-based systems, Chrome may be available in the repositories. However, for more control, consider installing the latest stable version directly from Google. - For other Linux distributions, you might need to manually download and install Chrome from the official Google website.

Installation of Node.js and npm (using nvm):

Node Version Manager (nvm) allows you to manage multiple Node.js versions on your system. Here's how to install and use it: Install nvm: curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash This command downloads and executes the nvm installation script. Follow the on-screen instructions to complete the setup. Verify nvm installation: source ~/.bashrc nvm -v The first command ensures your shell recognizes nvm commands. The second command checks if nvm is installed correctly. If successful, it should output the nvm version. Install a specific Node.js version: nvm install v20.12.0; This command installs the specified Node.js version and sets it as the active version. Verify Node.js and npm installation: node -v npm -v These commands should display the installed Node.js and npm versions.  

Installation of Redis server

Use the APT package manager to install Redis from the official Ubuntu repositories: sudo apt install redis-server  

Installation of Google Chrome:

- Download the Chrome DEB package from the Google website using the following command: wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb - Install the downloaded package: sudo dpkg -i google-chrome-stable_current_amd64.deb If installation errors occur due to missing dependencies, resolve them using: - sudo apt-get install -f Then, retry the installation command. - sudo dpkg -i google-chrome-stable_current_amd64.deb

Installation of Prerender.io:

To globally install Prerender.io, execute the following command: sudo npm install -g prerender

Configuration:

Create a config.json file with the following content: { "prerender": { "chromeLocation": "/usr/bin/google-chrome", "forwardHeaders": false, "whitelist": , "blacklist": , "removeScriptTags": true, "redisUrl": "redis://localhost:6379" } } Adjust the settings as necessary, such as specifying the path to your Google Chrome installation, defining whitelists and blacklists for URLs, removing script tags from pre-rendered HTML, and providing the connection details for your Redis server.

Starting the Server:

Initiate the Prerender.io server using the command: node server.js --config /path/to/config.json Replace /path/to/config.json with the actual location of your configuration file. The server.js File: Here's an example of how your server.js file might look: const prerender = require('prerender'); const server = prerender(); server.start(); This code initializes the Prerender.io server using the prerender package and starts it. Make sure to customize it according to your specific requirements and configurations.

Configuring PM2:

Create an ecosystem file (ecosystem.config.js) in your project directory with the following content: module.exports = { apps: };

Managing Prerender.io with PM2:

If PM2 is not installed globally, do so by running: sudo npm install -g pm2 Then, start Prerender.io using PM2: pm2 start ecosystem.config.js

Monitoring and Maintenance (Optional):

Search engines are designed to parse and index HTML content. Simple enough for static HTML pages or traditional MVC applications where HTML templates are rendered server-side. But what about modern web apps where HTML templates are rendered on the client?

Search bots are not going to run Javascript and will end up indexing your HTML templates instead. At best, they will be mostly empty. At worst, some otherwise hidden snippet of content will get inadvertently indexed.

<span ng-show="isEmpty()">No results</span>

The snippet above is only shown on empty search results but gets indexed indiscriminately and the Google crawler reports the entire page as a soft 404 crawl error. Ouch.

The problem in a nutshell - search engines index content, not apps.

Here are the two basic types of solutions to the Javascript SEO problem:

The most obvious is to have the server generate a custom static version of the page specifically targeted for crawlers. There are many ways of doing this, but they all end up forcing you to pretty much rewrite your entire app to do server-side generation. This is really a regression no matter how much clever code reuse you come up with. The static content can be a hidden part of the actual page or be served exclusively to search bots (the latter can regress to the much maligned practice of cloaking). What makes this approach most deficient though is that it cannot be applied universally to all web apps as it depends on each particular technology stack and requires constant maintenance.

The other approach is to simply run the Javascript code in a headless browser and give the resulting HTML snapshot of the page to the search bots. Let's look into this in more detail.

Generating HTML Snapshots in a Headless Browser

This approach is actually what the major search players are pushing for and you can read Google's specification here. Here's the basic idea.

The page to be indexed tells the Googlebot that it should be specially requested for indexing via a meta tag:

<meta name="fragment" content="!">

The bot will rewrite the URL by adding an additional query parameter _escaped_fragment_. The value of the paramter will be either empty or contain the escaped hashbang query if one is present. (I will not get into a discussion of the hash bang approach vs HTML5 pushState as it's orthogonal the problem at hand - just know that it works in both cases).

The server detects the presence of the special _escaped_fragment_ parameter and intercepts the request. Depending on your set up this can be done either in your Apache or Nginx configuration or directly in your ExpressJS middleware stack for example.

The URL of the interecepted request is then rewritten back to its original form and fetched from the server by a headless browser like PhantomJS and returned to the crawler.

Let's do it! - A Simple MEAN Stack Implementation

Various paid solutions abound - just do a simple search for "AngularJS SEO" or "Javascript SEO problem". Some of the more popular ones that I've looked at include prerender.io, brombone and SEO.js. Let's see how hard it is to write this from scratch for a MEAN (MongoDB, ExpressJS, AngularJS and Node) stack web app.

Let's start with detecting and forwarding _escaped_fragment_ requests. You can do this in your Apache or Nginx configuration, or directly in your ExpressJS middleware stack. Here's my Gist for the latter using http-proxy for request forwarding.

This middleware is very simple - it checks for the presence of the special query parameter and uses http-proxy to forward the request to our PhantomJS server. Here's a link to Gist containing to a simple PhantomJS setup as it too lengthy to include here in full:

 PhantomJS SEO HTML Snapshot Server

The PhantomJS server will rewrite the URL so we don't get into an infinite loop and fetch the request from our ExpressJS server. To run the server, install PhantomJS and invoke on our server:

npm install -g phantomjs phantomjs seo-server.js

One last bit here is how we signal PhantomJS that we are done loading the page so it knows its ready to take the snapshot. This is done via a special callback in our Angular app. Here's a simple module that registers a callback attached to the root scope - $scope.htmlReady(). This is best called on successful loading of all dynamic content.

angular.module('phantomjs-callback').run(['$rootScope', '$window', function($rootScope, $window) { // phantomjs callback when ready for SEO server to take HTML snapshot $rootScope.htmlReady = function() { if (typeof $window.callPhantom == 'function') { $rootScope.$evalAsync(function() { // fire after $digest setTimeout(function() { // fire after DOM rendering $window.callPhantom() }, 0) }) } } } ])

This is an update to my previous post: http://www.philipholly.com/post/64211668657/node-js-iisnode-prerender-io-phantom-phantomjs

Current hardware/software:

Windows 8.1

Visual Studio 2013

Git Extensions (including git bash)

Since I got a new work laptop, I needed to resetup my local websites.

This are the steps I went through to get a local instance of prerender.io up and running locally in a node.js server. Much easier than last time! I didn't need to install phantom.js server first. That simplified a lot. And maybe because I have Visual Studio 2013 instead of 2012 installed things just worked. I don't know.

Download node.js for windows

Run node.js windows installer

Let it add node to your path

Open Git Bash and navigate to the directory where you want to host your prerender server

run Git clone https://github.com/collectiveip/prerender.git. Instructions are here: https://prerender.io/server

Run npm install

I got the error: Npm install failed: pyhton 2.6 or 2.7 is required to build

Download python 2.7.6 Windows Installer: https://www.python.org/download/

Run python installer

Exit out of console

Open a new git bash window and navigate to your prerender project

Run npm install

This time it should work - I did get one warning C4506 but it doesn’t' look like it's a showstopper

That's it! This time I didn't have to install phantomjs server separately. Maybe last time I didn't but I didn't know that? I don't know.

Follow the instructions here: https://prerender.io/server to test if it worked