Christian's PC diary

Then I clicked the EC2 icon, and got this overview showing resources:

I supposed the Volumes was the culprit here, so I just clicked ‘Volumes’ and got into this page:

I selected the volume by clicking the little box in the left column, and then when to the ‘Actions’ menu and chose ‘Detach Volume’ and then clicking ‘yes’ in the popup asking if I’m sure.

I then chose ‘Detached Volumes’ in the dropdown menu, and the volume showed up in ‘available’ state. I select it again, and this time I chose ‘Delete Volume’ in the ‘Actions’ menu, and clicked ‘Yes, delete’ in the popup.

Now when going back to the EC2 overview, the resources show 0 Volumes (this took a minute or so to get updated though).

Installation of fcgiwrap: If you are on Debian you might just need to do:

$ apt-get install fcgiwrap

For RHEL/CentOS/etc there is no rpm ready. Instead, do the following:

$ cd /usr/local/src/ $ git clone git://github.com/gnosek/fcgiwrap.git $ cd fcgiwrap $ yum install fcgi-devel make automake autoconf $ autoreconf -i $ ./configure $ make $ make install

Running fcgiwrap with supervisord: Instead of manually starting fcgiwrap, it is a good idea to have some monitor software to start it. Here is how to do it with ‘supervisord’. (Install with ‘yum install supervisor’. Start with ‘service supervisord start’) Next check where fcgiwrap was installed:

$ which fcgiwrap /usr/local/sbin/fcgiwrap

Now create a file git_fcgiwrap.ini in /etc/supervisord.d/ with the following content:

[program:git_fcgiwrap] autorestart=true command = /usr/local/sbin/fcgiwrap -s tcp:127.0.0.1:9003 numprocs=1 numprocs_start=0 redirect_stderr=true stdout_logfile=/var/log/supervisor/%(program_name)s-%(process_num)01d.log user=git

And make supervisord read and start fcgiwrap by doing:

$ supervisorctl reread $ supervisorctl update

Installation of Gitweb:

$ apt-get install gitweb

or:

$ yum install gitweb

Configure Gitweb by setting the path to your Git repositories in /etc/gitweb.conf. I.e. add a line like this:

our $projectroot = “/home/git/”

Configuration of Nginx: And now finally, here is the configuration for Nginx:

server { server_name git.example.com; listen 80; access_log /var/log/nginx/gitweb.access.log; error_log /var/log/nginx/gitweb.error.log; charset utf-8; location /gitweb.cgi { fastcgi_pass 127.0.0.1:9003; fastcgi_param PATH_INFO $uri; fastcgi_param GITWEB_CONFIG /etc/gitweb.conf; fastcgi_param SCRIPT_FILENAME /var/www/git/gitweb.cgi; include fastcgi_params; } location / { root /var/www/git/; index gitweb.cgi; } }

Make sure that SCRIPT_FILENAME is set to the path of gitweb.cgi on your system, and also set root correspondingly.

In the below example we get the expected behaviour, 'Hello' is printed directly, and 'world' is printed after 5 seconds:

$ python -c "import time; print 'Hello'; time.sleep(5); print 'world'" Hello world

In the following example, we would expect the same behaviour, since 'grep o' should match both words, however, both 'Hello' and 'world' will be printed simultaneously after 5 seconds. This is because of buffering:

python -c "import time; print 'Hello'; time.sleep(5); print 'world'" | grep o Hello World

To prevent it, you can call python with '-u' (or use PYTHONUNBUFFERED=Y environment variable), which runs python unbuffered:

$ python -uc "import time; print 'Hello'; time.sleep(5); print 'world'" | grep o Hello world

You can also control flushing of output from within the script, like this:

$ python -c "import time,sys; print 'Hello'; sys.stdout.flush(); time.sleep(5); print 'world'" | grep o Hello world

 If you want to set stdout to be unbuffered from within the code, without doing flush all the time, do this:

$ python -c "import time,sys,os; sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0); print 'Hello'; time.sleep(5); print 'world'" | grep o Hello world

It might also depend on the command you are piping to whether output gets buffered:

$ python -uc "import time; print 'Hello'; time.sleep(5); print 'world'" | tail -f Hello world

This does not work since the tail is not applied until the output is finished, thus when EOF is received.

If you are using 'py2exe', you can package your application to be run unbuffered by adding the unbuffered option in setup.py:

options={"py2exe":{"unbuffered": True}}

Unbuffered output is probably something you want in most command-line utilities. But what should you do if you need to use an existing command which isn't unbuffered, and you don't want to modify it? There is a command named 'unbuffer' which takes care of this. The 'unbuffer' command is part of 'expect'. You can use it like this:

However, this doesn't work completely. "Hello world!" will be printed on [tty1], but the command 'cat /tmp/testfifo' will exit since EOF is received. Here is a trick to prevent this. Use 'tail -f':

[tty1]

$ tail -f /tmp/testfifo | cat - [tty2] $ echo "Hello world!" > /tmp/testfifo

Go to the backup directory and run the following commands (Replace 'example.tumblr.com' and '[email protected]'):

cd ~/backups/tumblr/example.tumblr.com/ echo "git log -1 -p | mail -s \"[tumblr-cli]\`git log -1 --format='%s' | cut -d ';' -f2\`\" [email protected]" >> .git/hooks/post-commit chmod a+x .git/hooks/post-commit

Make sure you can email from the server by doing:

echo "hello" | mail -s "Hello world" [email protected]

If this doesn't work you will need to set up emailing on your server. There are several help available by Googling. I used this.

Prerequisites:

A linux server (I use a virtual Centos 6 server on Linode, see my recommendation here)

Tumblr-cli installed on the linux server (see my post with instructions here)

Git installed on the linux server (Find it here)

Instructions:

Log in to the linux server.

Make a directory where you want the backups to be stored, e.g:

mkdir -p ~/backups/tumblr/example.tumblr.com/

Try making a backup manually using the command:

tumblr-cli-backup example.tumblr.com --git --root ~/backups/tumblr/example.tumblr.com/

and make sure that it works. I'm using virtualenv on my server, so my command would look a bit different:

~/.virtualenvs/env1/bin/python ~/.virtualenvs/env1/bin/tumblr-cli-backup example.tumblr.com --git --root ~/backups/tumblr/example.tumblr.com/

Open up the editor for the user's cron table: crontab -e

Add a row with the time (<minute> <hour> <day of month> <month> <day of week>) and the backup command as follows:

30 04 * * * tumblr-cli-backup example.tumblr.com --git --root ~/backups/tumblr/example.tumblr.com/

And this is it! Now your blog should be backed up every night at 04:30. Check that it is working by doing:

cd ~/backups/tumblr/example.tumblr.com/ git log

Linux has proven very well for such purposes, and SSH clients are available for all operating systems, so you don't need a Linux computer in order to get your own virtual Linux server. On the server you can run whatever you can think up, web servers, Git repositories, backup services and more. As you learn more, you will think up more and more use for it.

You can try a similar service as Linode for free using the Amazon EC2 free usage tier, however, their service seems to have been made by masters of business, making it very hard to navigate, and their price plans and vocabulary is very hard to grasp. As an example, here are all the services Amazon says I am signed up for, which are shown when logging in:

Amazon CloudFormation Amazon CloudFront Amazon CloudSearch Amazon CloudWatch Amazon DynamoDB Amazon Elastic Compute Cloud (EC2) Amazon Elastic MapReduce Amazon ElastiCache Amazon Glacier Amazon Mechanical Turk Amazon Relational Database Service (RDS) Amazon Route 53 Amazon Simple Email Service (SES) Amazon Simple Notification Service (SNS) Amazon Simple Queue Service (SQS) Amazon Simple Storage Service (S3) Amazon Simple Workflow Service (SWF) Amazon SimpleDB Amazon Virtual Private Cloud (VPC) Auto Scaling AWS Data Pipeline AWS Direct Connect AWS Elastic Beanstalk AWS Import/Export AWS Storage Gateway Elastic Block Store (EBS) Elastic Load Balancing Product Advertising API

Quite a list. When navigating their pages I might find what I am looking for, but then later I never find it back again. Its a feverish feeling, and still, after getting things up and running, using the server feels slow. I guess it makes more sense in a corporation where you get paid for the time spent with it, but otherwise it feels a bit frustrating.

I did finally succeed to create a free usage tier virtual server at Amazon, but I experienced unbearable lag (latency) when using the Amazon service (when trying from Canada) in comparison with Linode (On Linode it was unnoticeable). Now I tried it again from Sweden, and it felt better. When comparing ping times Amazon node answered in 114ms while Linode answered in 35ms, so Linode was still 3 times faster. When starting X applications, Linode feels much faster, but otherwise I didn't notice much difference.

With the Amazon free tier, the first 12 months are free under certain conditions (such as up to 750 machine hours a month) and above this it will cost. You need to register your credit card, and when googling the matter, you can see that a lot of people have ended up charged, when they thought they where using the free service. If something went wrong and a process spins away, it could be costly. It felt hard to survey what it would cost, and in the end it seemed to me that it will be less expensive with Linode, which costs $19.95 per month. Linode just feels solid and like it should be, and I've never had anything to complain about during my time of running a virtual Centos 6 server.

As a bonus, Linode now and then upgrades the service, and you get more performance for the same money, e.g. 14 December 2012 the storage increased by 20%.

Conclusion: If you absolutely want a free node, go with Amazon EC2 as it is the only alternative. If you find the complexity stimulating, you might also prefer Amazon, but otherwise I recommend Linode. It is totally worth the monthly fee.

Prerequisites:

A tumblr blog (Get one here)

Tumblr API keys (register here)

tumblr-cli 1.4 or later (See installation instructions here)

Git (If you want to make use of version control. Find it here)

First make sure you are authorized. Run:

tumblr-cli --blog example.tumblr.com --authorize

Now make a new directory where you want the backup files to be stored:

mkdir ~/tumblr-backups/ cd ~/tumblr-backups/

Make a backup (omit '--git' if you don't have Git):

$ tumblr-cli-backup --git example.tumblr.com Saved file './16663181489.example.taken-with-instagram-at-leons-full-service.meta'. Saved file './16663181489.example.taken-with-instagram-at-leons-full-service.html'. Saved file './16663181489.example.taken-with-instagram-at-leons-full-service.img/http___25.media.tumblr.com_tumblr_lyj8yrnc251qz5ul7o1_1280.jpg'. Saved file './12748481524.example.haterphile-taken-with-instagram-at-target.meta'. Saved file './12748481524.example.haterphile-taken-with-instagram-at-target.html'. Saved file './12748481524.example.haterphile-taken-with-instagram-at-target.img/http___25.media.tumblr.com_tumblr_lum52aDY151qz5ul7o1_1280.jpg'. Saved file './12746621592.example.l-taken-with-instagram.meta'. Saved file './12746621592.example.l-taken-with-instagram.html'. Saved file './12746621592.example.l-taken-with-instagram.img/http___25.media.tumblr.com_tumblr_lum36aNlfL1qz5ul7o1_1280.jpg'. Saved file './12745610341.example.taken-with-instagram-at-empire-state-south.meta'. Saved file './12745610341.example.taken-with-instagram-at-empire-state-south.html'. Saved file './12745610341.example.taken-with-instagram-at-empire-state-south.img/http___25.media.tumblr.com_tumblr_lum240cs5x1qz5ul7o1_1280.jpg'. Saved file './4012137257.example.upside-down-taken-with-instagram-at-home.meta'. Saved file './4012137257.example.upside-down-taken-with-instagram-at-home.html'. Saved file './4012137257.example.upside-down-taken-with-instagram-at-home.img/http___25.media.tumblr.com_tumblr_lifmbeDHb51qz5ul7o1_1280.jpg'. Saved file './4012075272.example.babies-john-s-lens-inas-1935-film-no-flash.meta'. Saved file './4012075272.example.babies-john-s-lens-inas-1935-film-no-flash.html'. Saved file './4012075272.example.babies-john-s-lens-inas-1935-film-no-flash.img/http___24.media.tumblr.com_tumblr_lifm6yN5Eo1qz5ul7o1_1280.jpg'. Saved file './4011000213.example.brannon-gets-home-taken-with-instagram-at-home.meta'. Saved file './4011000213.example.brannon-gets-home-taken-with-instagram-at-home.html'. Saved file './4011000213.example.brannon-gets-home-taken-with-instagram-at-home.img/http___25.media.tumblr.com_tumblr_lifjvw19p91qz5ul7o1_1280.jpg'. Saved file './4008972345.example.more-tests-taken-with-instagram-at-home.meta'. Saved file './4008972345.example.more-tests-taken-with-instagram-at-home.html'. Saved file './4008972345.example.more-tests-taken-with-instagram-at-home.img/http___24.media.tumblr.com_tumblr_life8cOtpM1qz5ul7o1_1280.jpg'. Saved file './4635069.example.multiparagraph-text-post.meta'. Saved file './4635069.example.multiparagraph-text-post.html'. Saved file './4627891.example.adult-swim-video-added-by-embed-tag-not.meta'. Saved file './4627756.example.youtube-video-add-by-embed-tag-automatically.meta'. Saved file './4627509.example.youtube-video-added-by-page-url.meta'. Saved file './4627288.example.a-chat-post.meta'. Saved file './4627198.example.link-to-my-real-tumblelog.meta'. Saved file './4627142.example.a-text-post.meta'. Saved file './4627142.example.a-text-post.html'. Saved file './4626997.example.large-photo-automatically-resized-to-the-template.meta'. Saved file './4626997.example.large-photo-automatically-resized-to-the-template.html'. Saved file './4626997.example.large-photo-automatically-resized-to-the-template.img/http___24.media.tumblr.com_4626997_500.jpg'. Saved file './4626666.example.small-photo-aligned-left-might-or-might-not.meta'. Saved file './4626666.example.small-photo-aligned-left-might-or-might-not.html'. Saved file './4626666.example.small-photo-aligned-left-might-or-might-not.img/http___25.media.tumblr.com_4626666_500.jpg'. Saved file './4626039.example.phasellus-faucibus-orci-ut-urna.meta'. Saved file './4626039.example.phasellus-faucibus-orci-ut-urna.html'. Saved file './4626002.example.sed-vestibulum-nulla-vitae-turpis-in-hac.meta'. Saved file './4626002.example.sed-vestibulum-nulla-vitae-turpis-in-hac.html'. Saved file './4625589.example.lorem-ipsum-dolor-sit-amet-consectetuer.meta'. Saved file './4625589.example.lorem-ipsum-dolor-sit-amet-consectetuer.html'.

Check the git log:

$ git log --oneline 4264970 '/home/chrisk/virtualenvs/virt1/bin/tumblr-cli-backup' '--git' 'example.tumblr.com'; 1327794725

First you will need to create an nRelate account, which is done here: https://partners.nrelate.com/registration/ Now log in using your new account, and choose Install->Tumblr

Enter your site's URL, e.g. 'http://example.tumblr.com/', and submit. Now you will asked where to place the related posts. I chose 'Posts Only', and got presented with the necessary HTML/CSS/JavaScript, and an instruction where to put it. Follow the instructions, surf to a post of your blog, and related posts should now appear.

To change style and other properties, go to 'Manage Settings'->'Tumblr', where you can choose thumbnail size, number of posts and much more. For the settings to take effect after you have saved them, you need to push 'Reindex Blog'. This takes a lot of time depending on the size of your blog (+10 minutes for mine).

You can see how the different styles look here: https://partners.nrelate.com/wp-content/themes/partners-main/images/style-gallery.html

Most important when preparing a Python package for distribution is to create the setup.py. This is the setup.py I used for tumblr-cli: (If you want to see how it relates to the code structure, look here: http://code.google.com/p/tumblr-cli/source/browse?repo=default)

from distutils.core import setup setup(name='tumblr-cli', description='Python Tumblr CLI', version='1.2', author='Christian Klöfver', author_email='[email protected]', url='http://code.google.com/p/tumblr-cli/', download_url='https://tumblr-cli.googlecode.com/files/tumblr-cli-1.2.tar.gz', install_requires=['argparse', 'oauth2', 'tumblr2'], packages=['tumblr_cli'], entry_points={'console_scripts': ['tumblr-cli = tumblr_cli:main', 'tumblr-cli-zim-tool = tumblr_cli.zim_tool:main']}, )

Most of the above variable names are self-explanatory. The 'url' is the homepage of the project. As you can see, I am hosting my project on Google Code. The 'download_url' points to the actual package, which I uploaded on the project homepage at Google Code.

To generate a package to upload, just run:

python setup.py sdist

After uploading the package to Google Code, you will want to register it at PyPI. This is simply done by running:

python setup.py register

Now it can be installed from PyPI with this command:

pip install tumblr-cli

In order for posting to Tumblr from Zim, you need to install 'tumblr-cli', a python package available on PyPI. First follow the instructions on this blog post, to install and authorize posting to your blog: http://www.klofver.eu/post/39789325684/tumblr-command-line-interface-tumblr-cli

You now need to add the Tumblr button in Zim, which is done by going to 'Tools'->'Custom Tools', clicking '+' and entering: Name: Tumblr Draft Description: Uploads the page to the Tumblr drafts Command: tumblr-cli-zim-tool example.tumblr.com %d %n Icon: <choose an icon> Check 'Show icon in toolbar' and click 'OK'.

See the picture to get the idea:

Since I am using python-virtualenv, the command I use looks slightly different, i.e: /home/chrisk/virtualenvs/my_virtual_env/bin/python /home/chrisk/virtualenvs/my_virtual_env/bin/tumblr-cli-zim-tool www.klofver.eu %d %n

You should now be able to publish drafts from Zim. If you want to make them public directly, add this argument '--param state=published' to the command. Or choose a state from these: published, draft, queue, private More parameters are specified here:

pip install tumblr-cli

You will now be able to run the command. Try '--help' first:

$ tumblr-cli --help

usage: tumblr-cli [-h] [--blog BLOG] [--authorize] [--post_text FILE]

                  [--title TITLE] [--param KEY=VAL] [--forceauth]

                  [--config FILE] [--pdb]

Tumblr Command Line Interface

optional arguments:

  -h, --help        show this help message and exit

  --blog BLOG       The blog to act on. E.g staff.tumblr.com or www.klofver.eu

  --authorize       Authorize to blog

  --post_text FILE  Post a text file to your blog

  --title TITLE     The title to be used in posts

  --param KEY=VAL   Extra parameter. See valid parameters and values here:

                    http://www.tumblr.com/api_docs

  --forceauth       Reauthorize to blog even if there already is a valid

                    access token.

  --config FILE     Configuration file

  --pdb             Puts you in pdb mode if any exceptions are raised.

To be able to make use of the command you first need to register an application, which is done here:  

http://www.tumblr.com/oauth/register

You will then get a consumer key and secret, which you will need when authorizing to a blog. For authorizing to a blog, run the following command and follow the instructions like this:

$ tumblr-cli --blog=www.klofver.eu --authorize

No value for consumer.key found.

What is the value for consumer.key? 8kafk9NNM5RpaLFxc3PM59Ef0JCmMLGKM

No value for consumer.secret found.

What is the value for consumer.secret? ZECx3UH0JIsASxeSIk1r5vkrtKY9peblgS

Visit: https://www.tumblr.com/oauth/authorize?oauth_token=Ef0JCmMLGKMBm7pyLRq1ZQwjLECZ8kafk9NN&oauth_callback=http%3A%2F%2Fwww.klofver.eu%2Ftumblr-cli

What is the oauth_verifier? Rq1ZQwjLECZ8kafk9NNM5RpaLFxc

After setting all the necessary values and fetching the oauth_verifier, you should now be authorized to make changes to the blog. Try posting a new draft using the following command:

echo "hello world" | tumblr-cli --post_text - --pdb --blog=example.tumblr.com

The post has now been sent. Check that the new post appears in the drafts folder of the blog you posted to.

Here is the Google Code page of the project: 

http://code.google.com/p/tumblr-cli/

And here is the PyPI package site: 

http://pypi.python.org/pypi/tumblr-cli/

The initial setup of a new Google Code project was all straight forward, except for choosing a license (which can be a jungle). I went for GPLv2. I choose Git for version control. The instructions for checking out the repository was pretty good. Just go to the "source" tab and follow the option 2 alternative, creating a ~/.netrc file , adding code.google.com and then giving it the right permissions as follows:

echo "machine code.google.com login [email protected] password DS67R34y4deWv" >> ~/.netrc

chmod go= ~/.netrc

I could then clone the empty repository like this:

git clone https://code.google.com/p/tumblr-cli/

I made some changes (Just created an empty Eclipse/Pydev project), added the files, committed, and pushed.

cd tumblr-cli

git add .

git commit

git push --all

I created a udev rule like this:

$ cat /etc/udev/rules.d/70-hdmi.rules KERNEL=="card0", ACTION=="change", RUN+="/usr/local/bin/my_screen_setup"

Here is the script that sets up the screens, and is run when ever the monitor is attached or detached:

$ cat /usr/local/bin/my_screen_setup #!/bin/sh DISPLAY=:0 /usr/bin/xrandr --output HDMI1 --primary --auto --output LVDS1 --right-of HDMI1

Root needs to be given access to the X server. This can be accomplished by putting the following row in your ~/.bash_profile file:

[chrisk@tmate optoreadrepo.git]$ cat ~/.bash xhost +si:localuser:root

This will be run when you log in. But if you want to try attaching/detaching right now, just run:

. ~/.bash_profile

or

xhost +si:localuser:root

This could have been a really interesting alternative, if it had worked. Unfortunately the result ended up very strange for me. I uploaded a piece of code like this: https://gist.github.com/4423995 And tried embedding it like this:

<script src="https://gist.github.com/4423995.js"></script>

And the result looks like this (A scrollbar for every line):

Trying a similar service, pastebin.com, was more successful:

But either way its a bit tedious for adding small code snippets, such as oneliners etc. Instead I tried Google's Prettify, following the instructions from this post: http://ghoti143.tumblr.com/post/1412901908/google-code-prettify-tumblr-love But unfortunately I didn't have much success with that one either. Code seemed to disappear, and just some got presented in the blog post. Probably it had to do with escaping of angle brackets, as the post mentions this. But having to escape those also interrupts the work flow, so I abandoned this method as well.

I tried a couple of more similar methods, but apart from not being easy to use, they also were making the page load a bit too slow. So instead I settled with just making a CCS style for code snippets using a fixed width font. Unfortunately indenting doesn't work well. So I think I'll use pastebin.com for longer scripts, and use the CSS style for shorter code snippets.

Here is the style sheet code:

div .code_snippet { font-family: monospace; font-size: 12px; color:#A40000; background-color:#E3E3E3; line-height: 15px; }

To use it, just surround the code with the following tags: <div class="code_snippet"></div> in your blog post (In HTML editor mode). You can see the result above, as this code are using the "code_snippet" class. I still think there should be better ways to accomplish this. There are a lot of browser addons for Tumblr, and also desktop applications, but so far I haven't found a reasonable editor where I can include properly indented and presented code snippets. May be it is possible to accomplish through the Tumblr API? If I get time, I think I'll have a look.

<!-- My CSS --> <style type="text/css" media="screen"> div .g-plusone, .twitter, .fb-like { font-size: 1px; display: inline-block; float: middle; } div .fb_reset { display: inline; } </style>

To add the buttons, make sure your <html> tag looks like this <html xmlns:fb="http://ogp.me/ns/fb#">, and add the following where the buttons should appear (I put it right before the tags "{block:IfDisqusShortname}{block:IndexPage}"):