Mahdi Yusuf

I just spent the better part of an hour trying to effectively install version of python that doesn't affect the rest of my system. This is usually pretty trivial effort according to most online articles. There are a few subtleties you might want to be aware of before you go and and well you know.

I am writing this mostly for myself so pardon the casual nature, and/or grammatical errors.

So you have a clean install of Mavericks, world is beautiful. You have a new beautiful new background and you have sudden inspiration to get some work done.

So many guides will tell you to do simply use system python which comes with easy_install and install pip with sudo.

Don't do this, For many reasons.

Homebrew always has the most recent version (currently 2.7.5).

Apple has made significant changes to its bundled Python, potentially resulting in hidden bugs.

Homebrew’s Python includes the latest Python package management tools: pip and Setuptools

First thing you do is install homebrew. Then do a little something like this.

brew install python

Once that is done, you will want to update your PATH variable to include the following directories in PATH

/usr/local:

/usr/local/bin:

You want to put local on PATH because it contains other important directories like lib and sbin.

What we are doing here is modifying the PATH variable which your system looks for commands, it will traverse the paths until it finds a match and use that. We want it to use copy of python in /usr/local

You can put these modifications in your .zshrc or .bashrc whichever floats your boat.

Now to test to see if you are using the correct python on your system (you have multiple copies now after the brew install) you want to you use:

which python

If you get /usr/bin/python you either didn't set the PATH properly or didn't reload either .zshrc or .bashrc (here is a hint: restart your terminal)

If you get a usr/local/bin/python you are winning. You will also want to make sure you are using the correct copy of pip on your system as well, use the same steps as above.

Never install anything with sudo. Don't listen to random READMEs online. You make the decision, my son.

Now you install things you want system wide without having to worry about breaking your system's copy of Python.

I am not a particularly disorganized person, but I am a person who does avoid organizing things. 

I recently saw the computer of a friend of mine who had everything organized meticulously. I mean to the point where when he works on something he has all related material, all organized by year and by type.

He doesn't share this information when works openly, but it was awe inspiring. 

This is incredible for two reasons he can look back on what he has done over the years and see how things are being done, and clearly see his improvement, plus all that nostalgia that comes with it. 

But, what really got me is that the mental structure he had for everything, there is very little for your mind to be worried about, no extra decisions to be made, just do what you set out to do. Beautiful. 

I have taken the stance for many years; just keep what i need, don't hold on to nothing, once it's served its purpose be done with it. no maintaining, no managing.

"Don't let yourself get attached to anything you are not willing to walk out on in 30 seconds flat if you feel the heat around the corner." --  Neil McCauley

I used to be the type of guy who use to categorize and label everything, then something just happened, got too busy or whatever excuse is handy and stopped doing it. 

Currently here is a snapshot of my downloads folder. 

This would have drove me nuts a few years ago, and everytime I dive in there I always cringe, but I made a decision, and its time to make another one.

This is mostly a note to myself that will probably come back and reference/update from time to time. 

Remember to Experiment

Do not forget this. Experimenting is how you find and create new things. Attempt new things, look for interesting problems. Plus its just plain old fun. Do the wacky stuff.

Think like a beginner

Take a fresh look at problems, don't go through your mental catalog of past problems. Each problem is unique and it has its own set of unique solutions. You might surprise yourself, with what you come up with.

Thought catalog

Remember to jot down recurring thoughts, you are likely not the only one who has them, there might be just something to them.

Take your time

This isn't a sprint it's a marathon, slow down silly rabbit.

Independent Study

Just learn for the fun of it, you never know what might come in handy. Derive your own conclusions, share with others. 

Be positive.

I recently released my latest project called Neckbeard Republic. It's a screen casting site where I cover various topics in Python. Invariably teaching the viewer something new or a new way to think about something.  

Our industry is an extremely fast moving one, and we spend a lot of time driving innovation. Its often hard to look to the right and see what our peers are working on.

As a curator of a newsletter, this is an excellent medium to receive weekly happenings in your industry of choice, without the torment and constant review of the same old sites. 

That being said I have compiled a list of newsletters you might be interested in. If I missed anything please hit me up and I will update it accordingly. 

As you know I run a weekly newsletter and we recently rolled out a jobs site, we've job postings from companies likes Path, Newsle, DataDog, and more. 

So somewhere in the middle of the night a timer hit zero and I was granted access to mailbox app.

Not complex enough feedproxy.google.com/~r/TheEndeavou…

— Mahdi Yusuf (@myusuf3) January 10, 2013

Developers fall to the same affliction when trying to solve problems. They try to make their solution as complex or as involved as possible; Amaze in their ability to keep things in context and their non-trivial solutions! Fight the stack!

Developers are the only ones who read programs, machines just execute them. Let's really take this point to heart. Readability over everything! How often do developers take into account the readers state of mind when writing code? How does that change when you are writing documentation? Both should emphasize clarity and simplicity.  

When I am most in sync with another developer is when I am reading his/her code I understand their mental process when trying to solve the problem. That combined with a clear problem statement and comments is what its all about!

Anytime I am reminded of elegant solution to a problem it is usually deemed elegant simply due its innate ability to solve the problem at hand. It's almost always easier to grasp and grok. Definitely easier to debug when you find a bug in your current implementation. 

I am glad this is becoming a habit. Writing end of year reviews gives me a chance to look back on the year. On the things I have accomplished and people I have met. Those of you interested in last year's review go here.  All of this work has been done outside of my 40-55 hour work week job. No excuses. So going into this year I set a few goals for myself, here they are:

Get better at design.

Learn a new language.

Do more open source.

Write fewer blog posts.  

Speak at a conference.

Focus (utterly failed here)

The Blog

A  lot of the work I do is shared through this blog. Going into this year I had a few goals in mind. First off, having longer editorials on things I found interesting or thoughts I find myself having constantly. I thought this might force me to take writing more seriously and provide deeper insight to people who follow my blog. In actuality, it caused me to write far less than what I would have liked.  In comparison to last year I have written 4 times less than the previous year. That being said I like the format of smaller more consumable articles as opposed to these giant monolithic blog posts, and in all honesty this form of content is better suited to my readers. In the new year, I will be regressing back to that format. I usually give yearly statistics on the traffic, and most popular posts. Here are some posts from this year that people seemed to have enjoyed.

Beautiful Tools

Hacking Rdio

Fasting and Programming

Reading Other People's Code (working on it I promise.)

Four Weeks With DuckDuckGo

The blog received a little over 250K uniques over the past year, ¼ of last year’s traffic. I am going to attribute that simply that I wrote less, if there is any correlation between me actually writing and traffic.  

Open Source

Another one of my big goals for the year was to do a lot more open source work and contribute to projects I get value from everyday. I started with Django. The community of core developers proved to be very welcoming and extremely helpful to people wanting to get in and contribute. That being said I wanted to provide a little more than a trivial fix so I picked off test discovery, a feature that has been discussed in previous conferences. After discussing my progress with other core developers and having my code reviewed, it was stopped from being merged in mainly because it was too close to the cut off for a dot release and some other concerns that were brought up later by other core developers. I can appreciate the caution and forethought of core developers but this might be due to the lack of new developers contributing and the general speed of iteration of Django itself, but that’s another story. So next I started working on a few projects of my own.

octogit

First of which being octogit, a command line tool for interacting with Github. I had started out this project before Github had created all the easy ways to create repositories through the web interface. I found it extremely annoying I had to paste remotes into my terminal in order to create a simple repository; it should be as simple as `git init` locally.  From that simple idea it branched into a tool for looking at issues in the command line and much more. This was pretty cool because people started submitting patches and contributing ideas and I got to see first hand what it was like to be an open source project maintainer and dealing with people’s amazing demands on my time :)

numbers.vim*

My next little project was a little vim plugin called numbers.vim.  After being turned onto vim by Mike. I noticed that vim had two types of numbering both relative and absolute. So for those of you who aren’t familiar with vim, absolute numbering is the numbering you usually see in all your editors. Relative numbering is relative to where your cursor is in the file so 0th line would be wherever your cursor is at the given time, this a very cool use in vim to allow you to manipulate blocks of code by saying something like delete the next 9 lines for example. This is a plugin that provides intelligent switching of the numbering types based on which mode you are in. If you are a vim fiend you should definitely give it a look. Its also included by default in the popular spf13 vim distribution. * I have since abandoned using vim as my main editor, project will be maintained though so no worries.

Speaking/Design

I have always wanted to get better at speaking. I like to challenge and push myself so it was on my todo list this year. I usually don’t get too nervous when I am speaking. My real worry, however, is that people don't find the topic interesting or the talk boring or uninspired. Despite these concerns, I think I did well and there is definitely room for improvement. I had an awesome experience at Pycon CA. Those of you interested in getting feedback on speaking or slide I would definitely give http://speakup.io/ a look. 

Aside from speaking I put a huge amount of effort into the design of my slides, and here is where I pushed my obsession with fonts to the limits. I played around with a bunch of fonts, colors palettes, slide placement I found this to be extremely fun and a worthwhile experience! There are tons of resources for idea and inspiration on SpeakerDeck for interested parties. 

<h2>Pycoder’s Weekly</h2><p>Probably the project that is the most fun to work on and honestly the most humbling one this year. Mike and I started <a href="http://pycoders.com/" data-mce-href="http://pycoders.com/">Pycoder’s Weekly</a> a newsletter for python developers by python developers. To date we have sent out 45 issues and currently have over 7500 subscribers in the Python community.<br /><br />Honestly the thing for me that is the most fun is all the people we get to interact with on a weekly basis and all the cool projects we get to see. Also planning and talking about what we can do next to grow the community forward. This has taught me tons about what the Python community finds interesting as well as what is required to cultivate and grow any community.<br /><br />We have exciting stuff planned in the new year. <br /><br /></p><h2>Conclusion</h2><p>When I was first stubbing out this blog post I was really unimpressed with the amount of work I did this year in my free time. I am usually quite hard on myself and often leads me to be unhappy with myself from time to time usually never lasts long though, but after writing this I am content. :)<br /><br />Few things I would like to see improve includes:<br /><br /></p><ul><li>Limiting  commitments on my time and focus.</li><li>I would also like to see myself start a product or project that I could turn into a business, I will be hopefully writing about this a lot in the new year.</li></ul><p><br />Also to anyone starting out share your ideas, talk to people, be kind. Do awesome things. Don’t worry so much about failure and what others think. :)</p>

Pycoder’s Weekly

Probably the project that is the most fun to work on and honestly the most humbling one this year. Mike and I started Pycoder’s Weekly a newsletter for python developers by python developers. To date we have sent out 45 issues and currently have over 7500 subscribers in the Python community. Honestly the thing for me that is the most fun is all the people we get to interact with on a weekly basis and all the cool projects we get to see. Also planning and talking about what we can do next to grow the community forward. This has taught me tons about what the Python community finds interesting as well as what is required to cultivate and grow any community. We have exciting stuff planned in the new year.

Conclusion

When I was first stubbing out this blog post I was really unimpressed with the amount of work I did this year in my free time. I am usually quite hard on myself and often leads me to be unhappy with myself from time to time usually never lasts long though, but after writing this I am content. :) Few things I would like to see improve includes:

Limiting  commitments on my time and focus.

I would also like to see myself start a product or project that I could turn into a business, I will be hopefully writing about this a lot in the new year.

Also to anyone starting out share your ideas, talk to people, be kind. Do awesome things. Don’t worry so much about failure and what others think. :) Set some goals, work towards them!

I am writing this mostly out of frustration with developers. Hopefully this can shine some rules of conduct when discussing software on a medium like Github.

Being a developer, I understand that there are many things combating beautifully crafted software but thats the beauty of open source:

Your time.

Your release dates.

Your requirements.

Another beauty of open source is collaboration and collective effort to craft better software .

Responsibility

In my mind a group of people behind a project have a certain level of responsibility to the people who use their software, to those who potentially would like to contribute, and to those who contribute.

Contributors provide additions that improve the project as a whole and improve the use of the software for its intended audience and within the maintainer's vision.

Maintainers

Recently I have seen my fair share of this type of action when it comes to pull requests being closed without dicussion or comment.

Your display of autocracy is fully within your right, but please don't spare us of your banality. Say something, Why isn't it a good idea? What can be improved for inclusion?

I would love to see this taken a step forward where maintainers outline what they think will be needed in the next release. Organized in issues with milestones, Github has all that is needed. So that contributors have clear places to contribute and to look for improvement.

Now an obvious counter argument is to say I can do it better myself, and it takes as much time to write the issue than to write the code. Not true, Linus does this quite effectively with the kernel albeit way more harsh with poor pull requests they still recieve his feedback.

Contributors

You have two options:

Maintain your own fork of the project.

Contribute to the collective.

Most people think if my change isn't going to be accepted why should I improve the project. I don't know maybe to solve problems you are having? If enough people voice that same concern maybe its worth contributing back. If not maintain your own fork. :)

That being said if maintainer has some stylistic choices for his project or general feedback be an adult and take the criticism lightly.

If it's a big feature you want to write; talk to the maintainer first see if they have thought about it. Before you go and write all this code and have it rejected for reason X.

Also there are many ways to contribute to a project that many people overlook but are very much needed.

Testing (I have never seen tests get rejected)

Documentation.

Reviewing other Pull Requests.

Raising reproducible issues.

Conclusion

As maintainer we should be willing to accept and list where our software can be improved.

I really enjoy reading other people's code, it allows you see how others solve common problems, and sometimes even provide parables on what you shouldn't do with yours. It also teaches the much needed skill of reading other people's code.  Unfortunately there isn't a convenient and fun way of reading other people’s code. 

Now, ask yourself two questions:

how many hours a week you spend writing code?  

how many hours do you spend reading code written by someone you don't know or have access to?

The brave might even ask:

how much of that code am I confident I understand, and could contribute to?

Many people avoid reading code simply because they hate being outside of their comfort zone. Documentation is an effective use of time if it does what it says. Often times documentation is outdated and unreliable and maybe even lies. Only thing you can be sure of is the source. 

Entire businesses have been built on the idea of sharing and writing code, but not reading.

Reading

What about reading?  I would argue reading code is as integral to writing as listening is to talking. Everyone likes to talk/write its a form of expression; the art behind the craft, but neither can be appreciated without their counterparts listening/reading. 

If I were to hand you a book and ask you to take me to where you would start reading, you would flip past the acknowledgements, table of contents,  and other credits and go to the first page, or chapter one. Fantastic. 

Now if I were to hand you a stack of code I printed out could you tell me where to start reading it as easily? Could you break the code up into chapters as easily? Should you the reader have to? What if it doesn’t have a main method? :)

Before I answer that, I would like to tell you a story.

When I first started out, I loved writing things from scratch. The clarity of how everything worked together made me feel powerful. I understood everything because I wrote it, I didn’t need to read it, because I wrote it.  NIH (not invented here) is basically a thing because rewriting is actually easier than reading the source in developers minds; developers develop and winners win.  

Why would anyone rewrite a program so they could add some functionality, when they could just as easily patch the already existing program to do what they want?

This involves the following things:

Obtaining the source

Reading & Understanding the source

Contributing/Patching

Testing (hopefully)

Sharing

Most would agree step 2 is probably what they are trying to avoid for the most part which brings us back to our original question. 

Where is the first chapter or first page in a piece of software? In my opinion its where the writing begins. This shares many analogies with book writing in general. The architecture of a book may not change (beginning, middle, and end) but they share many parallels to software development having  an end goal in mind at the start. 

So to answer the question:

Where do you begin reading software? It’s where the writer starting writing it. Thats irregardless of it actually does anything. Does an unfinished story say anything? You can still read it right?

Rainman

That being said, today I am going to start working on Rainman. A tool that will use git repositories (to start out) to allow the replay of git commits and diffs in a sane manner providing transparency of the software writing process.

So you might be thinking that commits are too large to be useful. That might largely be the case, thats why I also intend to provide a recording tool that will allow developers to record their software development process and allow a replay that is more granular and indicative of the development process. 

This is still in the inception phase and welcome feedback and ideas in the form of issues here. 

I'm not sure how many of you know but for the past 10 days I have been fasting. Fasting you say? No, I am not protesting the demolition of a national park (not that I wouldn't I love parks).

For those of you not familiar with Ramadan it's a holy for the month for muslims where they fast from sunrise to sunset. That's not entirely true, its technically between the prayers that occur very close to those times. This fast includes food and drink!

I also thought it might be interesting to share my perspective as software developer practicing this fast.

Lunar vs. Solar Calendars.

That being said I won't go too deep into the religious details but it is something that I practice every year.

It occurs at different times in the solar calendar due to the fact that Ramadan is a month in a lunar calendar which consists of 354.37 days compared to the solar calendar's 365 days. Therefore the times in which a fast occurs shifts throughout the  solar year. 

For example this year it landed in probably the highest number of daylight hours possible, and by next year it will only shift backwards ~two weeks relative to where it was this year on the solar calendar. So fasting is going to get harder before it gets easier. 

Winter is coming.

Preparation. 

So you know you going to be fasting tomorrow, literally the entire day of no food and drink. What would you eat to make that day go by easier? 

Okay so I can eat right up until sunrise? So that means wake up just before sunrise and eat like a boss. 

When is sunrise? 4:09 am EST for where I live.

To say the least thats a annoyance; especially when you have to be up in 4 hours to go to work.

Usually for me I find it really hard to eat anything after waking up from sleep, so I drink as much water possible. Hunger usually abides( by abides I mean you get used to it) after the first couple days. The real struggle is thirst. You end up getting really parched.  

Drink lots of water. Food too if you can stomach it.

Work.

I am a full time developer, which means I spend most of my days looking at a terminal and writing code. I am also no stranger to extreme measures to increase productivity and focus.  

Let's just say fasting isn't one of them. All of my co-workers; I have worked with past and present have been extremely understanding and respectful of the fast.

Also as a by-product of this kindness they have coined the term: Ramadan code. 

But all joking aside; I have to say it is incredible how worse off you are when you aren't able to eat(or drink) and trying to do work that is involved. For example today I was in some code that I wrote literally a little over a month ago without difficultly and for the life of me I was unable to write something very similar without doing it over 6 times.

Your ability to focus and keep things in context is severely diminished; that bundled with the shadow of the man you used to be can be quite the 1, 2 punch to your ego and more than your daily recommended source of frustration.

In conclusion, this has taught me a few things:

Eat well.

Exercise.

My yearly dose of humility is only months away.

Don't skip meals to save time. 5 good hours are better than 10 bad ones. 

Hopefully this was insightful; if you have any questions please hit me up.

The more OAuth providers there are, the more inconsistency there is. Not following the specification puts an increased and unnecessary burden on the developer.

In the past couple of weeks I have coordinated and authenticated with more than my fair share of OAuth providers. That being said, you should be thinking "Hey, isn't OAuth a specification?" Once you do one you should be simply swapping out keys and urls and you should be good to go right? So wrong.

Follow the specification and be consistent.

This article is meant to let the OAuth providers know some of the pain points of dealing with them as providers and steps they could take to make interacting with their API a joy.

API Documentation

Like with all documentation, I need to experience joy in the first 15 minutes otherwise your documentation has failed. Clearly outlining API endpoints and how I can get the data I need is crucial. Also another big win that some APIs provide (like Foursquare and Google) is an API explorer which shows you how to make a request and what to expect as a response. Amazing for getting familiarized with a new API.

Don't point me at another services documentation (looking at you _Tumblr_) write your own.

Another thing that enrages me to no end is the lack of consistency. If you are returning JSON formatted data in some places and returning the body of a HTML page in others you are doing it wrong. This requires me to do two things: process which ones return JSON and which return HTML data I have to parse. Pick a way to represent the information returned from your API and be consistent. But I believe the specification has something to say about this.

Error pages should also return data in a consistent manner.

Logos

Now a lot of OAuth providers don't provide appropriate graphics for using their services. I want to use your service to authenticate my users, but you aren't providing proper graphics for my site.

You would think that OAuth providers would want to have the most amazing logo being sent out to the public. Twitter does an excellent job of this, it has also increased Twitter's presence as an OAuth provider on tons of sites and applications. Do the graphics have something to do with it? Maybe!

Screencap of Twitter authentication buttons

Let's be honest we have all seen this. Make the effort provide logos, buttons and anything else I may need to properly identify your service in the authentication process.

Required Fields and UIDs

Now when a user signs up websites try to make it as easiest as possible to increase user sign up and adoption. Then you provide an API that exposes this information and has data gaps because you wanted to avoid having the user type in one more field during sign up. The API should drive those requirements not the other way around.

Another important thing I find that some providers are ignoring (like Tumblr) is a unique id for your users. It is absolutely essential for each user to have UID that doesn't change and is consistent for the life of the account.

Conclusion

If you run a business and decide to become an OAUTH provider or support an API it is essential that you put as much emphasis on this portion of the product as you would on any other core feature. It portrays your attention to detail as well as your ability to build a cohesive product. This should translate to an eager development community working with your tools rather than writing snarky blog posts complaining about them. :)

Follow the specification.

Provide clear and concise documentation.

Provide graphics.

The following post was a post I made couple of days ago. Then took it down once I realized the affect it had on the Rdio team. 

The Rdio team has taken steps to rate limit the downloading in the way described in the following post. Which means its still possible but extremely slow to download. I am sure there are other illegal ways to get music you want. 

On a personal note, I learned quite a bit from this experience. 

It all started innocently enough. Last day of work before a two week vacation. I use Rdio all the time for music while programming. Embarrassingly so in fact.

So in preparation for my trip, I was trying to download some of the talks from DjangoCon Europe for my flight for offline enjoyment. Using one of these pieces of software I noticed that it was able to download the current song I was listening to on Rdio. 

So 3 flights later and 4 days without reliable internet later. I managed to reproduce the fluke that I randomly ran into. 

Here is how I did it. 

Poking around.

Once I saw that software I was using to offline videos was downloading Rdio songs. I popped over into the inspector and took a look at the network tab and saw that they are sending a file called full-192.mp3 which means full song at a bitrate of 192K. 

Hmm. Okay.

Rolling up my Sleeves

So I decided to crack out some tools. First I needed to get a better understanding of the communication between Rdio's servers and my machine. 

Enter mitmproxy, which is a man-in-the-middle proxy for HTTP. This gave me a bit more of an understanding of the traffic Rdio is sending around and eventually allowed me to script a way to download full mp3 versions of any song on their service. 

On to the details. 

As you can see in the image above; We were also able to capture the GET request being sent for the full-192.mp3 which returns the entire song being requested.  That's all good. 

Whats awesome is the response below. 

Yep, the entire content of the full-192.mp3!!

Once you have reached this point a little (under 10 lines) script is all that is needed to download the full mp3 to your computer. 

Hopefully the Rdio Team gets a handle on this soon. 

Edit: Same bug occurs in their "native" application as well on OSX.