Exploring...

On MacOS:

#install using the command line

curl "https://awscli.amazonaws.com/AWSCLIV2.pkg" -o "AWSCLIV2.pkg" sudo installer -pkg AWSCLIV2.pkg -target /

#verify installation

aws --version

#configure AWS CLI

aws configure

You'll need to enter:

AWS Access Key ID: [your access key]

AWS Secret Access Key: [your secret key]

Default region name: [e.g., us-east-1]

Default output format: [json is recommended]

#test configuration

For some reason, I do not use the builtin logging module in Python as often as I think I should. I am not sure why. I have used logging several times before, yet it has not become part of my standard tool kit. My first suspicion is that I am just being lazy and print statements are so easy. But now as I wade back into the waters, I am starting to remember that logging uses some form of magic that I never took the time to get comfortable with.

The magic appears when you use logging across multiple modules. Suppose you have an app called app.py that loads a function from a module as in the following code:

# app.py from my_module import my_func my_func()

7 # my_module.py import logging logger = logging.getLogger(__name__)

def my_func():    logger.error('message from mod 1')    logger.debug('a debugging message')

If you run app.py you will get the error:

No handlers could be found for logger "my_module"

But add an import in app.py and suddenly the error in my_module.py goes away:

? 1 2 3 4 5 6 # app.py from my_module import my_func import logging

logging.error('A message from app.py') my_func()

I lied, you also have to have line 5.

I cannot think of another case where an import in one module effects another module. That's why I did not like using logger. I never got comfortable with how it worked across modules. But magic is also powerful. Time to learn it. Before we go any further, you should read the basic tutorial in the Python docs.

The questions we are going to address are: how does logging come to life? how does it stay alive when execution passes to a new module? The answer is that once the logger is configured, it becomes part of the Python interpreter process that is executing your code. It effectively becomes global.

Part of the confusion also comes from how easy it is to configure logging. As shown in the snippet above, simply importing logging and logging a message auto-magically configures a logger for your entire project. Convenient, but un-nerving if you do not know its happening.

Here is a simple example of a more explicitly configured logger:

? 1 2 3 4 5 6 7 # app.py from my_module import my_func import logging logging.basicConfig(filename='example.log',level=logging.DEBUG)

logging.error('A message from app.py') my_func()

The logger in the previous example wrote to the console. This logger writes to a file and changes the default level. The amazing thing here is that the log messages in mod1.py also show up in this file.

This also explains why the line:

? 1 logger = logging.getLogger(__name__)

shows up in so many modules. This line causes the logging messages to be prepended with the module name. Pretty helpful when you have 47 different modules writing to the same log file.

I think that does it for me. With these mysteries cleared up, the rest of the Python docs are pretty straight forward.

But since we have gotten this far, I want to share a debugging trick. Often when I am debugging a module, I run it as __main__ along with some setup code. Here is how to make the logger work with __main__ and when it is called as part of a larger app:

# mod1.py import logging logger = logging.getLogger(__name__)

def mod1():    logger.error('message from mod 1')    logger.debug('a debugging message')

if __name__=='__main__':    logging.basicConfig(level=logging.DEBUG)    mod1()

grep -E --color=always 'svm|vmx' /proc/cpuinfo

yum groupinstall "Virtualisation Tools" "Virtualization Platform"

yum install python-virtinst

chkconfig libvirtd on

service libvirtd start

service libvirtd status

virsh -c qemu:///system list

A very good answer to the question "What is a metaclass in python".

"If you want to please only the critics, don't play too loud, too soft, too fast or too slow."

The last time we used our flip video camera was Dec, last year. Yesterday we decided to charge it and record some videos with it. But it refused to charge!

Doug said worst case we can use 3 AAA batteries instead. But I decided to try again this morning. It was still connected to the charger on the wall. I tried to turn it on, but it briefly showed the "flip video camera" picture, and beeped, then showed "connected", then went off. 

I thought it was some connection problem. Since it was on the wall, and the screen of the camera was facing the wall, I moved it to a power strip, so I could flip the strip, and see the screen while it was charging. Again and again, it would automatically show the "flip video camera" pic, then "connected", beep, and go off.  I was stubbornly turning it on again and again, then I noticed a new pic, which was the low battery signal. That was hope!

But except for the new low battery signal, it goes in loop again. And there was no sign showing it was charging. So I tried turning it on again, and again. Finally I saw the sign of charging. But it goes off again.

I resorted to google, found an article about taking the battery out of the camera at different times between or after, or before plugging it in the USB port and unplugging it. I was about to try that, another article caught my eye, saying a long time not used flip came would charge off a USB port on a computer monitor better, and the author tried on a Mac USB and worked, which didn't work on the AC charger.

Learning is not compulsory, neither is survival.

Once again I had trouble configuring the authoritative servers for DNSSEC testing. I am recording the lessons learned here, so I can come back when I have problems again. 

This might not apply to other DNS servers. The one I used is ANS/P. 

Zones configured:

. (root)

.                SOA     x.root-servers xnstld.verisign-grs.com 2012021949 1800 900 604800 86400

com, net, edu, org, test1, test2

dnssectest.com, dnssectest.net, dnssectest.edu, dnssectest.org, dnssectest.test1, dnssectest.test2

Basically, don't set up any zones for root-servers.net, or gtld-server.net. Use A record only for those, e.g. 

I am writing this post to summarize the 1.5 weeks I spent on performance tuning of the new machines we got. 

Before that, I had got the best performance of our multi-threaded DNS cache resolver from another set of machines, which were borrowed from another team. Then we purchased these new machines, which have 20 CPU cores (versus 16 in the other set of machines), 132G memory (versus 64G), the same NIC (10gbE). The operating system is CentOS 6.5 on both sets of machines. But the performance I got from the new machines were miserable (1/5 of the best performance).

This is the first time ever I have been involved on machine performance tuning. So I had no clue where to start. 

I started by inspecting all the network card settings, receive & transmit buffer setting, and the rx-flow-hash setting. Except for the new machines have a newer version of network card firmware, I didn't find anything different. I asked our IT guys if it was possible to downgrade the firmware version, and was told NO. 

Then I started looking at the CPU settings. Man there is so much difference there. First I noticed the CPUs in the new machines have different sets of flags from the cores in the old machines. So I asked the IT guy if we could change the flags there. I was told NO again, unless we are the OEM (of course we are not). At this point, I felt I was pretty stupid anyway for asking these two questions. Although I couldn't really find online whether those changes could be made. The IT guy got a little impatient too I think, because he sent out an email to all the engineers with the RedHat performance tuning book link, and some tap scripts.

The other difference I found between the two sets of machines is that there is only one NUMA node in the new machines with all the 40 cores (20 physical cores hyperthreaded), while there are two NUMA nodes in the old machines with 32 cores (16 physical cores hyperthreaded). There are two sockets in the new machines too, but why only one NUMA node? I was really suspecting that was the whole reason, and started looking into how to add another NUMA node. Unfortunately there was little information I could find online on that topic. Then my coworker informed me a tool that does exactly the same thing I was testing but only without I/O. I used that tool, and got better performance in the new machines, which ruled out the CPU performance issue. 

Now I had the idea that I should do some bench marking, which should have been the first thing to start with. 

The network performance tool I picked is iperf, because it can test the performance with multiple parallel streams, which is the case in our engine. The results showed exactly the same network performance between the two sets of machines. 

Then I started benchmarking everything else with sysbench. It's a wonderful tool because 1)It can test multiple things, including CPU, memory, thread and file IO. 2) it supported multithreading as well. 

So I tested all of those. The results showed slightly slower multithread file I/O, CPU and memory, but better threads and 1-thread file I/O. Since our internal tool showed better performance in the new machine without the I/O, I wasn't too concerned about the slightly slower CPU and memory. But now what is the issue?

I turned to the system tap scripts our IT guy threw me and found the nettop.stp. I had to install the debug package of the matching linux kernel to use the script, which I did on both sets of the machines, when I found out on the old machine the kernel is a slightly newer version. A big difference I found is that in the new machines, ksoftirqd was doing all of the packet receiving, while our engine is doing the transmitting; but on the old machines, our engine is also doing the transmitting, but it also does some transmitting, and the rest of the transmitting is done by a process called swapper (with a process ID 0). 

I had no idea why that was. So I started suspecting it is because of the newer version of kernel on the old machine. There was me again quickly filed a ticket asking the IT guy to update the new machine kernel to the same. But then, I saw actually that was the only machine in the whole set of old machines that had the newer kernel version. All the other ones have the same version as what's in the new machines, and I did get the same performance from the other old machines as well.     OK, that's probably not it. Apologize again Tod. 

Keep looking, I found another possible culprit - irqbalance. It was not even installed in the new machines!

So the kernel itself has an irq balancer. But it is not intelligent enough to distribute the workload to the cores effectively to achieve the best performance. irqbalance could help. So I installed it on the new machines. But I didn't see much performance improvement. 

Then I turned it off on the old machines - performance dropped by a half! I started it again, performance didn't come back!    !!!

Now I was super confused. Maybe there was some configuration I need to do to irqbalance? I had an 1-1 with my manager, who suggested I ask the performance engineer who might know more about the configurations on those old machines, where I might be able to get a quick answer from. 

A quick asking-around told me nobody has touched any performance tuning on those old machines. So I was left with no clue again... I was pretty tired and decided to take a break. 

The next morning I came in and started looking at the all the configurations. I found I tricked myself by not setting the rx-flow-hash on the old machine when I restarted them, and that was the whole reason why the performance didn't come back after I started irqbalance. 

When you play with python a lot, you'll find easy_install is such a useful tool, which is a python module bundled with setuptools that lets you automatically download, build, install, and manage Python packages. Here is a very good webpage showing how to install it on windows:

Just read a really good perl tutorial. Posting it here so I will remember it later... (you know, bookmarks don't work across computers)

install SIP:

python configure.py -d /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages --arch=i386

make

make install

install PyQt4:

python configure.py -d /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages --use-arch=i386

make

Bugs in PyQt are being found and fixed constantly. So when you find the bug is fixed in a version and are not sure which version you are using, here it is:

This is very helpful.

http://zetcode.com/tutorials/pyqt4/

Something worth mentioning here is this menubar tutorial, which took me quite a while to debug. 

# menubar.py import sys from PyQt4 import QtGui, QtCore class MainWindow(QtGui.QMainWindow): def __init__(self): QtGui.QMainWindow.__init__(self) self.resize(250, 150) self.setWindowTitle('menubar') exit = QtGui.QAction(QtGui.QIcon('icons/exit.png'), 'Exit', self) exit.setShortcut('Ctrl+Q') exit.setStatusTip('Exit application') self.connect(exit, QtCore.SIGNAL('triggered()'), QtCore.SLOT('close()')) self.statusBar() menubar = self.menuBar() file = menubar.addMenu('&File') file.addAction(exit) app = QtGui.QApplication(sys.argv) main = MainWindow() main.show() sys.exit(app.exec_())

Apparently on Mac "exit", no matter lower case or upper case or a combination of lower and upper, is treated as a reserved menu action, because there is a default "Quit Python" menu with "Command+Q" as the shortcut. So when you execute this code, "File" menu will simply not show up.

nosetests -w simple tests/test_stuff.py:test_b

Note that by default nose only looks for doctests in non-test code. You can add --doctest-tests to the command line to search for doctests in your test code as well.

nosetests -a \!will_fail

pickle is for serializing some object (or objects) as a single bytestream in a file.

shelve builds on top of pickle implements a serialization dictionary where objects are pickled as well, but is also associated with a key (some string), so you can load your shelved data file and access your pickled objects via keys. This could be more convenient were you to be serializing many objects.