computernotes

Apple changed the firewall from ipfw to pf sometime around 10.8. Sometimes, one must block specific ports or perform more advanced firewalling than ApplicationFirewall allows. For example, I sought to block portmap reflection attacks on UDP 111, which cannot be done using ApplicationFirewall or the Security system preference.

I found this discussion of pf on OSX very helpful in building these rules: http://blog.scottlowe.org/2013/05/15/using-pf-on-os-x-mountain-lion/

The steps:

1) Create your own pf.conf file. We create a different pf.conf file rather than modifying the system one because Apple has a habit of overwriting system files in updates.

sudo cp /etc/pf.conf /etc/my.pf.conf sudo nano /etc/my.pf.conf

2) Add these lines to the bottom:

anchor "myrules" load anchor "myrules" from "/etc/pf.anchors/my.rules"

3) Then save and exit. Note that OSX has a tendency to replace " with “ which will cause issues. Quotes must be regular quotes and not the fancy OSX ones.

4) Now, create your ruleset. This is a basic set that allows SSH, HTTPD and AFP only. Create this file in /etc/pf.anchors/my.rules

sudo nano /etc/pf.anchors/my.rules

set block-policy drop set fingerprints "/etc/pf.os" set ruleset-optimization basic set skip on lo0 # Scrub incoming packets scrub in all no-df # Antispoof antispoof log quick for { lo0 en0 en2 }

# Block to/from illegal destinations or sources block in log quick from no-route to any

# Block by default block in log

# Block to/from illegal destinations or sources block in log quick from no-route to any

# Block portmap reflection attacks block in quick proto udp from any to port 111

# Allow critical system traffic pass in quick inet proto udp from any port 67 to any port 68

# allow ssh, http, AFP pass in quick inet proto tcp from any to port 22 pass in quick inet proto tcp from any to port 80 pass in quick inet proto tcp from any to port 548

# Allow outgoing traffic pass out inet proto tcp from any to any keep state pass out inet proto udp from any to any keep state

5) Test the ruleset. You should see your rules printed. If there are any syntax errors or etc here, fix them.

sudo pfctl -vnf /etc/my.pf.conf

6) Create a new LaunchDaemon plist file for the firewall to load on boot

sudo nano /Library/LaunchDaemons/my.pf.plist

7) Fill the file with this XML, and note you may have to modify the location of your custom pf config file generated in step 1 if you used something other than /etc/my.pf.conf

<?xml version="1.0" encoding="UTF-8" ?> <!DOCTYPE plist PUBLIC "-//Apple Computer/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict>        <key>Label</key>        <string>my.pf</string>        <key>Program</key>        <string>/sbin/pfctl</string>        <key>ProgramArguments</key>        <array>                <string>/sbin/pfctl</string>                <string>-e</string>                <string>-f</string>                <string>/etc/my.pf.conf</string>        </array>        <key>RunAtLoad</key>        <true/>        <key>ServiceDescription</key>        <string>FreeBSD Packet Filter (pf) daemon</string>    <key>StandardErrorPath</key>        <string>/var/log/pf.log</string>        <key>StandardOutPath</key>        <string>/var/log/pf.log</string> </dict> </plist>

8) Change ownership of the file to root

sudo chown root /Library/LaunchDaemons/my.pf.plist

9) Fire it up with launchctl!

sudo launchctl load /Library/LaunchDaemons/my.pf.plist

10) And verify it’s loaded:

launchctl list | grep my

Python’s map and the multiprocessing module’s Pool.map both only permit functions of one argument. When using map, one can use a lambda function to get around this. However, these cannot be pickled for Pool.map.

It’s easy. Just use itertools izip and repeat functions to make an iterable tuple of arguments and then unpack them inside the function. For example: 

pool.map(func, izip(range(10), repeat(argb), repeat(argc), repeat(argd))

is equivalent to calling func(1:10, argb, argc, argd) across the pool. However, you have to rewrite the function to only take one argument and then unpack the tuple like follows: 

def func(args):    i = args[0]    argb = args[1]    argc = args[2]    argd = args[3]    # do stuff 

Now can use this function inside pool.map. Can alternatively define a wrapper function, as in: 

def pool_wrapper_func(args):    func(*args)

Just upgraded to OSX El Crapitan Capitan and naturally it fried my apache2 config. Really missing the interactive diff replacement in gentoo upgrades. Anyway.

To troubleshoot, look in /var/log/system.log. If there are error messages about apache (e.g. “Service exited with abnormal code: 1“) then run:

sudo apachectl stop sudo apachectl

The latter should tell you what the error message is. Resolve the error, and then

sudo apachectl start

In my case, there were multiple errors.

First, Apple decided to obliterate my httpd.conf and replace it with a new default, moving mine to /etc/apache2/httpd.conf~previous (which incidentally is a terrible name).

Second, there is a syntax error in the shipped httpd-mpm.conf, located in /etc/apache2/extra. Fix is by deleting the “The accept serialization lock file“ section as here: http://apple.stackexchange.com/questions/211015/el-capitan-apache-error-message-ah00526

Third, some of my old virtual server configs had excluded options with “-” and other options without “+”. this is apparently disallowed now and if any options start with “-” then the others must start with “+” or “-”.

Fourth, mod_ssl and many other modules I needed were commented out in the new Apple provided httpd.conf. 

Fifth, it seems as though my vhosts only worked when I placed them in /etc/apache2/extra/httpd-vhosts.conf and then uncommented the include for this in httpd.conf.

Sixth, since this involves upgrading from Apache 2.2 to 2.4, you have to change Directory permission directives from:

Order allow, deny Allow from all

to

Require all granted

(see here: http://httpd.apache.org/docs/trunk/upgrading.html)

Sometimes in OSX the kernel_task process is pegged at 100% CPU and/or the machine becomes sluggish to respond. If you look on the Apple forums most of the reports are about overheating laptops, but it can be caused by many issues.

If kernel_task is using high CPU/memory, you can use osxkerneltrace to figure out what system calls are leading to the CPU usage. In my case on my “trashcan” MacPro the kernel_task usage appears to be caused by my Drobo (not the first issue I have had with this RAID).

If the system is running slowly in general or you want more info about active I/O, I recommend Brendan Gregg’s awesome description of DTrace scripts for OSX: http://dtrace.org/blogs/brendan/2011/10/10/top-10-dtrace-scripts-for-mac-os-x/

I normally create a bowtie2 index of the repeatmasker repeat elements (LINE/SINE retrotransposons, Alu elements, etc) and map RNA-seq reads onto this before mapping on to the transcriptome. I do this primarily for two reasons: first, the huge number of reads from repeat elements may spuriously multimap onto other similar sequences, distorting gene expression, and second, when you are multimapping reads onto a transcriptome the repeat elements all multimap which ends up taking forever.

Requirements:

A FASTA file of the whole genome downloaded from UCSC or Ensembl or Illumina iGenomes

bedtools, bowtie2, a computer

Steps:

1) Download a .bed file of the repeatmasker track from UCSC using the Table Browser

2) If using ensembl, rename chromosomes:

sed -e "s/^chr//g" rmsk_hg38.bed > rmsk_ensnames_hg38.bed

3) Create a fasta file from the RMSK bed:

bedtools getfasta -name -fi hg38.fa -bed rmsk_ensnames_hg38.bed -fo test.fa

4) Generate the bowtie2 index:

bowtie2-build -o 1 rmsk_hg38.fa rmsk_hg38

Note: if you see a bunch of warnings about “just gaps” it’s because some input sequences are only N’s. To ignore these, append “2> /dev/null” to the command.

5) Optional: create a GTF from the BED (requires UCSC Kent tools)

El Capitan changes the owner of /usr/local to root:admin. This won’t work for homebrew.  Execute the following to change the owner back to your user to fix:

sudo chown -R $(whoami) /usr/local

The steps to download a video from facebook without visiting one of those sites that make you feel like your identity was just stolen.

1) Go to the video page (url should look like https://www.facebook.com/USERNAME/videos/BUNCHOFNUMBERS/)

2) View source

3) Search for “mp4″ and copy the line containing the first hit

4) Paste this to a text editor

5) Trim it down to something that starts with https and ends with “oe” and a bunch of numbers, with lots of unicode in between (things that look like \u002522)

6) Paste into a unicode converter, e.g. second box on https://www.branah.com/unicode-converter; convert from unicode to unicode text

7) Paste into a raw URL decoder, e.g. http://www.cafewebmaster.com/online_tools/rawurldecode

8) Now you will have a proper URL but the “/” will be escaped with “\/”, so delete all the “\” in the URL

9) Make sure the URL ends with “oe=SOMENUMBERS”

10) Use wget or etc to download the source mp4

I post things here about using computers that I have found useful and don’t want to have to figure out again. Maybe it will help you too.