#readthedocs

Let’s be clear. Nobody likes writing documentation. Writing good documentation is also hard. Making it look visually pleasing can be even more challenging. I’ve been involved with a project aimed to make user documentation easy to consume but also easy for anyone to contribute to. My north star for good documentation is the Ansible documentation. It’s visually very pleasing, easy to find stuff and it’s all there. Google does a damn good job indexing it so whatever you need is three keywords away. The project at hand is not capable of leveraging some of magic sauces behind the Ansible documentation project but I found a middle ground that completely blew my away. From here on forward this is the toolchain I will recommend for all and any documentation project: MkDocs and GitHub Pages.

Requirements

The bulk of our starting source docs today is all written in markdown and we were not interested in converting to a new “language” or learn a new one. It has to be version controlled, human readable and reviewable markdown. It disqualifies a bunch of different popular tools out there, but hear me out her. So, GitHub is an excellent place to store, version and review markdown files, let's start there.

Next, there needs to be a way to edit and review the markdown rendering locally (or remotely) efficiently before submitting pull requests. It should not be counter-productive with multiple tools, renders and manual refresh/walkthrough navigation to get visual feedback. As we sniffed around other successful documentation projects we learned about MkDocs as we investigated the capabilities of Read The Docs. Little did we know that MkDocs lets you render and edit docs locally very efficiently, it’s widely used, extensible and looks beautiful out-of-the-box. Just add markdown!

Also, MkDocs can deploy directly to GitHub Pages by putting the rendered output in a separate branch and all of a sudden you have everything in one place. That alone makes it very convenient as we don’t have to interact with separate services to host the documentation. One might think we’re done here but it leaves one big gap in the solution, that is the reviewing of pull requests part. In the event of a pull request, the person who merges to master needs to render the pages after the merge. You may quickly resort to readthedocs.org for this reason but what if I told you that there is a GitHub Action available that does this for you already? That changes the game. Full control end-to-end through GitHub. Let's do it!

Hello World

Since it wasn’t glaringly obvious to me on how to piece everything together, I thought I share my findings in this blog. Let’s walk through a Hello World example where we start with nothing.

First, create a new empty GitHub repo and clone it (you need to create your own repo as my demo repo won't work).

$ git clone https://github.com/drajen/hello-docs Cloning into 'hello-docs'...

Next, we need to install mkdocs if you haven’t already. Acquiring Python and pip is beyond the scope of this tutorial.

$ sudo pip install mkdocs

Change dir into the hello-docs directory and run:

$ cd hello-docs $ mkdocs new . INFO - Writing config file: ./mkdocs.yml INFO - Writing initial docs: ./docs/index.md

The mkdocs command creates the docs directory, this is where your source markdown lives. A skeleton index.md is populated with some MkDocs metadata. There’s also a starter mkdocs.yml file that allows you to configure your project. I want to use the Read The Docs theme, so, let’s configure that:

$ echo 'theme: readthedocs' >> mkdocs.yml

Next, we want to visually inspect what the documentation looks like:

$ mkdocs serve INFO - Building documentation... INFO - Cleaning site directory [I 200311 16:29:05 server:296] Serving on http://127.0.0.1:8000 [I 200311 16:29:05 handlers:62] Start watching changes [I 200311 16:29:05 handlers:64] Start detecting changes

Browsing to http://127.0.0.1:8000/ should now present the following website:

In an attempt to try illustrate how the local editing works, I generated a GIF from a screen capture. Simply edit text in your favorite editor (vi) and hit :w. The content will automatically be rebuilt and reloaded based on your markdown edits.

This is awesome!

Publish!

Not quite done yet. To demonstrate the next steps, we need to publish our site. Let’s add site/ (where the local build lives) to .gitignore and push our content.

$ echo 'site/' >>.gitignore $ git add . $ git commit -a -m 'Initial hack...' $ git push origin master

Next, have MkDocs publish to the gh-pages branch.

$ mkdocs gh-deploy INFO - Cleaning site directory INFO - Building documentation to directory: /Users/mmattsson/code/hello-docs/site WARNING - Version check skipped: No version specificed in previous deployment. INFO - Copying '/Users/mmattsson/code/hello-docs/site' to 'gh-pages' branch and pushing to GitHub. INFO - Your documentation should shortly be available at: https://drajen.github.io/hello-docs/

Visiting the URL MkDocs spits out above should be rendered in a few moments. It’s possible to tweak the URL by setting a custom domain for GitHub Pages under the repository settings. You’ll need a DNS CNAME pointing to <user/org>.github.io for that to work properly.

Action!

Now the skeleton is published. How do we accept pull requests and have the gh-pages branch rebuilt on merge to master? This GitHub Action does that exact job for you.

It only works reliably with personal tokens. A token is generated on your user account and a secret is part of the repository settings.

So, in the middle bar you’ll find an “Actions” tab, create a new workflow and paste in the YAML from the Deploy MkDocs GitHub Action. Don’t forget to change the token attribute!

Now, this GitHub Action will run for each merge to master, that makes it easy to accept pull requests for markdown. If big navigational changes is in the PR, it could make sense to clone the pull request and render the branch locally. That will allow visual inspection of the navigation and check for errors in the MkDocs build log.

New Post - Boring Technology Interview with @ericholscher about @readthedocs, open source funding and boring technology.

Boring, stable technology is king. If you’re running a huge site, you need things to work and work reliably. In this interview, my brother who also happens to run ReadTheDocs talks about sustainable funding for open source projects, getting people to work to support them without it, and boring technology. [soundcloud url="https://api.soundcloud.com/tracks/274627159"…

The pyzqm tutorial is very good for learning the basics of how to use zmq. I've learned a lot from reading it. Now I'm reading the PyZMQ API documentation because I feel that investing some time in learning the finer apsects of ZMQ is worth the effort. I'm discovering some handy tricks that I thought I'd share on my blog.

You know how you use a Context instance to create sockets? Well, the Context class has a very nice method for closing all sockets that were created by that context. Here's how you do it.

Here's a program that creates two servers and two clients that sends a message every second. After three seconds a timer kicks in and closes all

import zmq from time import sleep from multiprocessing import Process from threading import Timer, Thread context = zmq.Context() def server(name, port): server = context.socket(zmq.ROUTER) server.bind('tcp://*:{}'.format(port)) while True: print('{}: {}'.format(name, server.recv_multipart())) def client(name, port): client = context.socket(zmq.DEALER) client.connect('tcp://localhost:{}'.format(port)) while True: client.send_multipart(["", "From {} with love".format(name)]) sleep(1) def destroy(): context.destroy() if __name__ == "__main__": Process(target = server, args = ('server1', 5563)).start() Process(target = server, args = ('server2', 5564)).start() Process(target = client, args = ('client1', 5563)).start() Process(target = client, args = ('client2', 5564)).start() Timer(3, destroy).start()

However, this program won't work as intended. The clients will keep sending and the servers will keep printing what they receie. Why is this? Because we created the servers and clients as processes! When you create a process a copy is made of all the objects in scope. So the context object that the timer closed was never used to create any sockets! Aha, basic lesson in processes. Actually I made exactly that mistake :).

So let's create the servers and clients as threads instead:

if __name__ == "__main__": Thread(target = server, args = ('server1', 5563)).start() Thread(target = server, args = ('server2', 5564)).start() Thread(target = client, args = ('client1', 5563)).start() Thread(target = client, args = ('client2', 5564)).start() Timer(3, destroy).start()

This time the message flow stops, but not in a very nice way. The server threads throw an ContextTerminated exception, the clients throw ZMQError: Socket operation on non-socket exception, and then the program hangs. It's actually written in the docs:

destroy involves calling zmq_close(), which is NOT threadsafe. If there are active sockets in other threads, this must not be called.

So finally my conclusion is that context.destroy() is mainly useful when you have a single thread using many sockets. There are plenty of use cases for that. Here's a simple example where a single thread has two threads talking to each other (pretty useless, but just for the purpose of demonstation :)

Ok, now, create software with such kick-ass interface that it ships without docs. And, open out the APIs already!

While at that, ensure two parts to it:

Documentation minus fluff plus extensive parameter definitions.

A ton of testable APIs!

Only then shall we be superheroes as cool as The Avengers and cronies or more! Am yet to reach stage-2, but shouldn’t hesitate to add some Swagger soon ;) 

API documentation and static site generation is a kind of child’s play with Sphinx - why hop when you can fly! Right? Learning to write the doc source code in reStructuredText (reST) is some amount of work, but the kind of satisfaction for something so lean and easy is totally worth it.

How to...

My method is essentially, do it and repeat it.

(One time) - Run sphinx-quickstart command to generate the framework - makefile, conf.py, _index.rst, etc.

Manually write the source in reST.

(Ongoing) - Add necessary overrides to theme_overrides.css and conf.py

Run sphinx test build on my totally crappy Dell (ubuntu 14.04) work laptop.

Check source into Git when satisfied.

Pull the source into the staging VM.

Make HTML.

Preview and fix again. Checkin if any changes to source.

Pull the source in the Prod box and build HTML again.

This automatically publishes on our Website. Ta da!

... aaaaaand, repeat when necessary. 

Sphinx CSS Overrides

Why and how I chose Sphinx for API documentation is a story for another day. For today, am going to focus on some extensive CSS overrides and workarounds that should allow you to customize the theme to satisfaction and get your doc Website rolling. You may find some of these on the Web on various Sphinx or technical forums, but many of them are new.

Am on an older version of the sphinx_rtd_theme. The new one comes with some of these resolved, but I like the older CSS for some basic reasons. I have a separate theme_overrides.css to accommodate for this.

CSS Override - Table Width, Text Wrap, Scroll

The default sphinx_rtd_theme seems to make all tables render with scroll bars, no text wrap. My requirement was to wrap text for all normal tables, and have scroll for really wide tables that go out of view.

/* override table width restrictions */

    .wy-table-responsive table td, .wy-table-responsive table th {         white-space: normal !important;     }

    .wy-table-responsive {         margin-bottom: 24px;         max-width: 100%;         overflow: auto !important;     }

/* wide table scroll-bar */

    table.scrollwide {         display: block;         width: 700px;         background-color: #E0;         overflow: scroll; !important     }        table.scrollwide td {         white-space: nowrap;     }

Navigation Sidebar Scrolling

In the older theme, the navigation sidebar had no vertical scroll bar for long ToCs. So, if you’d scroll to read the HTML page content, the ToC may scroll out of view.

/* override navigation sidebar out-of-view on page scrolling */

    .wy-nav-side {

        position: fixed;         padding-bottom: 2em;         width: 300px;         overflow-x: hidden;         overflow-y: scroll;         min-height: 100%;         background: #343131;         z-index: 200;

    }

Navigation Header Background Color

The hex code for the default blue is #2980B9. You can change it to anything you’d like.

/* changed side navigation bg color */

    .wy-side-nav-search {

         background-color: #005387 !important;

    }

Navigation Menu Overrider: Margins, Text Color, Background Color

These are minor, misc. changes.

/* override navigation menu caption and link text colors */

    .wy-menu-vertical .caption-text {         color : #7e7e7e;     }     .wy-menu-vertical a {          color : #f0f0f0;     }

/* override caption margins */

    .wy-menu-vertical p.caption {          margin-top: 20px;          margin-bottom: 5px;

         /*  margin-left: 10px; */

     }

Override Home Icon

This was, by far, the trickiest for me to figure out. My excuse? Am not a CSS person. Tool help from our UI folks here and done!

/* hide icon-home */

    .fa-home:before, .icon-home:before {

         content: none;     }

/* override fa-home with logo */

    .wy-side-nav-search>a:hover {          background-image: url("logo.png") !important;          background-size: 30px 30px;          padding-left: 35px;          padding-bottom: 10px;          padding-top: 6px;          background-repeat: no-repeat;     }

    .wy-side-nav-search>a {          background-image: url("logo.png") !important;          background-size: 30px 30px;          padding-left: 35px;          padding-bottom: 10px;          padding-top: 6px;          background-repeat: no-repeat;     }

Conf.py Changes

Very simple. Ensure that you’ve pointed to the theme you’re using and then add the following references.

Favicon setting

My favicon (the home icon too) is in the static folder.

html_static_path = ['static', 'images'] html_favicon = 'favicon.ico'

Overrides CSS Reference

def setup(app):   app.add_javascript("custom.js")   app.add_stylesheet("theme_overrides.css") 

If you’ve created separate override files, no issues, just replicate the same reference structure pointing to your filename.

You can see my W.I.P. here.

As we continue to enhance Phalcon with new functionality and features, we have migrated all of our documentation on Read the Docs, a service that hosts documentation for open source projects such as ours.

Utilizing this service allows for:

Easy exporting the documentation to other formats like PDF or ePub

Managing the documentation for each release, independent of one another

Supporting additional languages (other than English) for the documentation

Consistency checking, ensuring that there are no broken links or missing images and much more.

This move required a total rewrite of the documentation, from PHP and HTML, to reStructuredText, a format that is independent of the code base and allows generation of the documentation to different formats.

Read the Docs is not yet considered the official documentation, since it is not complete. It does however serve as a quick reference for development. Our goal is to have the documentation as complete as possible prior to our next major release 0.5.x.

A new github repository has been created to manage the documentation. Every time anyone commits a change, Read the Docs automatically downloads and rebuilds the documentation, ensuring that it is up to date with the latest developments.

What's next?

If you have some extra time, we invite you to take a look at the new documentation and share your thoughts with us. If you find a typo, want to improve a section, wish to see something new that we have not covered, feel free to fork the repository and send us the changes via pull request.