Flickr Code

When I poke my head above my monitor at FlickrHQ, I can see a lot of headphones. Ever wonder what inspires us to write code (besides fried chicken sandwiches and cocktails)? Well I did a little snooping in the playlists of a few Flickreenos:

Zack

Top Tracks:

Fitz & The Tantrums - MoneyGrabber

Ear Infections mix - Goldenchyld

Live In San Diego - Goldenchyld

Little Birdy - Brother

Parliament - Red Hot Mama

Ross

Top Tracks:

David Gray – Flame Turns Blue

Kings of Convenience – Singing Softly to Me

Grizzly Bear – Southern Point

The Roots – The Next Movement

Iron & Wine – Jezebel

Trevor

Top Tracks:

Crawler - Cosmic Tide

Quest for Fire - Next to the Fire

Bison BC - Die of Devotion

Radiohead - Little by Little

DMX Krew - Do It All Nite

Scott

Top Tracks:

Wax Tailor – No Pity

Ray Charles – Eleanor Rigby

Steve Martin – How to Meet a Girl

KRS-One – MC's Act Like They Don't Know

Quantic – Snakes in the Grass

Oh, and I suppose I should share mine too:

•  Blueprint Car Crash – Gun Moll

•  Little Dragon – Come Home

•  Angus & Julia Stone – Black Crow

•  Plants and Animals – Celebration

•  of Montreal – Coquet Coquette

  Find even more of what inspires us on our FlickrHQ Last.Fm Group

During the redesign of our photo page, we created a YUI3 API  module, used for all Flickr API transactions. This module has been designed to load component modules as needed, which makes for a speedy initial load. For example, the YUI3 IO module, which handles XMLHttpRequests in the YUI3 library, is loaded only if writing to the Flickr API. Read requests are all handled by the Get Module, which is included in the core of YUI3. A byproduct of this approach is that read requests can be made cross domain, without a proxy.

Reading from Flickr

Simply include gallery-flickr-api in your requirement definition. Then use the Y.flickrAPI.callMethod function, passing the Flickr API method name and callback as arguments. Have a look at the YUI3 Gallery page for more information about using gallery modules. Read requests can be made cross-domain without a proxy, since the request and response will be loaded as a script resource instead of using XMLHttpRequest.

YUI().use('gallery-flickr-api', function(Y) { Y.flickrAPI.callMethod('flickr.panda.getPhotos', { api_key: 'your_api_key', panda_name: 'ling ling', extras: 'license, date_taken, owner_name', per_page: 50, page:1 }, { success : function(response_object) { //Success Code }, failure : function(response_object) { //Failure Code } }); });

Writing to Flickr

Okay, writing to Flickr API (i.e. post requests) is a little trickier. You will need to make sure that your API key is enabled to read and write. As soon as you make the first API request to write, the YUI3 IO module will be loaded (if it is not already). It is important to note that write requests can only be done on the same domain, which is the standard security restriction of the XMLHttpRequest for browsers. POST requests to the Flickr API, will require some sort of proxy which will accept the post request on your domain and handle the cross-domain communication with api.flickr.com

YUI().use('gallery-flickr-api', function(Y) { Y.flickrAPI.callMethod('flickr.favorites.add', { api_key: 'your_api_key', photo_id: a_photo_id }, { success : function(response_object) { //Success Code }, failure : function(response_object) { //Failure Code } }); });

The Joy of Popup Windows As you've probably noticed, we've been working on the login process here at Flickr. In an effort to make this flow as painless as possible we've moved to a "contextual" login, which is to say, logging in doesn't require you to leave the page you are looking at. We accomplished this by using a popup window. We initially considered using an in-page modal dialog box instead of a popup, but quickly dismissed this approach for two security reasons. Firstly, to prevent phishing and cross site attacks it's very important that login forms are not posted cross domain (Yahoo!'s Membership Platform authentication happens on yahoo.com, not flickr.com). Secondly the URL of the login page should never be hidden from the user. An in-page modal fell foul of both of these concerns. A more suitable solution would be the popup browser window, once a favorite of advertisers.

Our initial approach

Revisiting our turn of the millennium Javascript for opening windows with the BOM, things looked simple enough: call window.open, make sure the window opened, then set a timeout to see if the window was closed:

function waitForWindowClose() { if (web1999Window && web1999Window.closed) { // the pop-up was closed, done with auth etc. window.location.reload(); } else { // check again in a moment setTimeout(waitForWindowClose, 1000); } }//cute function name by Scott Schiller function webCirca1999(url) { try { web1999Window = window.open(url,'newWindow','width=640,height=650'); } catch(e) { //handle blocked popup... } try { //this is to check if it really opened if (web1999Window.focus) { web1999Window.focus(); } } catch(ee) { // also handle failure } waitForWindowClose(); return false; }

Unfortunately Yahoo!'s new user signup process closed our popup and opened a new one of their own. This triggered our waitForWindowClose{} function, web1999Window && web1999Window.closed evaluated to true, and the Flickr page refreshed, even though the user hadn't logged in yet. The next thing we tried was watching window focus. When the window blurred after clicking on the login popup, we'd know the popup was open, and when focus came back we'd check the user's login credentials. But focus tracking also had its problems. Some browsers didn't always report the window blur event when a popup was open, resulting in inconsistent behavior. It was also possible for the user to accidentally focus back on the window and kill the flow. We tried several ways to track focus, and although behavior got better, it was never 100% reliable. For a feature like login that was obviously a problem.

The Cookie Solution

Ultimately we settled on a solution that forgot about popup tracking and window focus and instead concentrated on checking for the user's login cookies. The code below polls to see if the login cookie is set, then makes an AJAX request to verify it, then refreshes the Flickr page:

function pollCookie(){ if(document.cookie.match(sessionRegex)){ authInProgress = false; checkAuth(); }else{ Y.later(20, this); } }

An embarrassingly long time ago we released the first public version of the Flickr Shapefiles. What does this have to do with Captain America and a cat? Nothing, really. Anyway, we haven't completely forgotten about shapefiles and have finally gotten around to generating a new batch (read about Alpha Shapes to find out how it's done). When Aaron did the first run we had somewhere around ninety million (90M) geotagged photos. Today we have over one hundred and ninety million (190M) and that number is growing rapidly. Of course lots of those will fall within the boundaries of the existing shapes and won't give us any new information, but some of them will improve the boundaries of old shapes, and others will help create new shapes where there weren't any before. Version 1 of the dataset had shapes for around one hundred and eighty thousand (180K) WOE IDs, and now we have shapes for roughly two hundred and seventy thousand (270K) WOE IDs. Woo. The dataset is available for download today, available for use under the Creative Commons Zero Waiver:

http://www.flickr.com/services/shapefiles/2.0/

Little Johnny JSON

Originally we provided the full dataset in our own home-grown XML format because, well, it seemed like a good idea. For version two we're releasing the shapes in GeoJSON format. We think this is a Good Thing because unlike our  old XML format, at least one other person in the world already knows how to read and write GeoJSON. For example, Our friends over at Stamen Design and SimpleGeo have created a ridiculously easy-to-use JavaScript library called Polymaps which of course reads GeoJSON out of the box. With a few lines of JavaScript you can render the Flickr shapefiles and start using them without all that pesky XML parsing stuff: Or if GeoJSON doesn't suit you you can use a free tool like ogr2ogr to convert it to something that does.

Layers

(photo by doug88888)

The GeoJSON format allows grouping of features (and their related geometries) into FeatureCollection objects. A FeatureCollection seems to be roughly equivalent to a layer in a typical GIS, or a placetype in WhereOnEarth-speak. To make the dataset a little easier to manage we decided to break the shapes up into FeatureCollections based on placetype; one each for continents, countries, regions (states), counties, localities (cities), and neighbourhoods. Each of these is its own GeoJSON file in the dataset. Here's an example of what one of our GeoJSON objects looks like:

{ "type": "FeatureCollection", "name": "Flickr Shapes Public Dataset 2.0 - Regions", "features": [ { "type": "Feature", "id": 2344541, "properties": { "woe_id": 2344541, "place_id": "Cxf0SmObApi9R9T8", "place_type": "region", "place_type_id": 8, "label": "Barbuda, AG, Antigua and Barbuda", }, "geometry": { "type": "MultiPolygon", "created": 1292444482, "alpha": 0.03, "points": 118, "edges": 17, "is_donuthole": 0, "link": { "href": "http://farm6.static.flickr.com/5209/shapefiles/2344541_20101215_724c4cae47.tar.gz", }, "bbox": [-62.314453125,17.086019515991,-61.690979003906,17.93692779541], "coordinates": [ [ [[-61.739044,17.587740], [-61.735268,17.546171], [-61.690979,17.426649], [-61.765137,17.413546] ... etc

A file is a single FeatureCollection object which holds an array of Features, which each hold a Geometry which is a MultiPolygon, which holds an array of Polygons which in turn each consist of an array of LinearRings. Got it?

You've Been Superseded

We've also included a separate file/layer called flickr_shapes_superseded.geojson. This is a FeatureCollection that consists of all the WOE IDs that have been "superseded". Occasionally (too often?) a WOE ID needs to be retired and replaced with a new one. We keep up to date with these and are always reverse-geocoding against the latest WOE IDs. However, there are plenty of old photos (and Flickr shapes) that have been assigned to one of these old IDs, and we have shapes for them (currently a little more than nine thousand). A simple solution might be to just re-assign these photos to the new WOE IDs (when a WOE ID is retired its replacement is specified), or even just re-run the reverse-geocoding process. This may not be what the owner of the photo wants; it might come out with a different result than they had at first (which they may have been perfectly happy with), if the size and location of the WOE rectangle changed (which it probably did). So it's a problem without a clear solution. And since we have data for these old WOE IDs we've included their associated shapes in the dataset. In most cases there will be shapes corresponding to the new WOE IDs that will over time become more and more accurate as more photos get uploaded and end up being assigned to the new WOE IDs. But you can have the old ones too.

WTF

In July 2010 we launched a new photo page that displays photos at up to 640 pixels on the largest side. In the months leading up to this launch we did a lot of work to support generating this new size, along with making our support of the Large image size more consistent.

Medium 640

We started adding support for the new Medium 640 size(_z suffix) on March 31st 2010. We generated a Medium 640 size for all photos uploaded after that date. Over the next couple months after that we found and corrected instances where we were not fully supporting this size. We display the Medium 640 size on the Photo, Photostream and All Sizes pages.

Dynamically Generated Images

If a photo was updated/uploaded before March 31st 2010 then we do not have a Medium 640 size on file and will try to dynamically generate it. We can do this if a large version of the photo exists, or if the original is a jpeg that has not been rotated(when you see an image URL that ends in _z.jpg?zz=1, the zz=1 indicates we can generate from an original). By doing this we make a lot of old photos display at 640 on the new Photo, All Sizes and Photostream pages. And every time we generate a Medium 640 image it gets cached, so we don't have to generate it again for a while. If we can't generate the image dynamically, we fall back to displaying the Medium 500. Generating photos dynamically needs to be happen fast, so we don't try to generate a Medium 640 from the original if the photo has been rotated or needs to be converted from a non-jpeg format.

Larges For All

We wished we had generated a Large size in the past for all photos uploaded, it would have made it simpler to dynamically generate the Medium 640 size. Before May 25th 2010 we only generated Larges for photos uploaded greater than 1280 in size, now we generate a Large no matter what size is uploaded. This is great for certain types of photos, for example we can now display 800x600 photos(common to some camera phones) as a Large in Lightbox or on the All Sizes page. So, a Large photo will be 1024 pixels in size on the longest side, or the same size as the original if something smaller than 1024 pixels was uploaded(we never scale a photo larger, an original photo of 800x600 will have a Large that is 800x600). You can expect the Large to always exist for newly uploaded photos.

Using the API

Who are you, and what do you?

I'm Stephen Woods and I'm a frontend engineer at Flickr. Along with the rest of our team I am responsible for the bits that make the UI work, which mostly means html, css and javascript. So I write code, but I also make sure the design of the site looks and works the way the designers want it to.

What hardware are you using?

In terms of computers I have 15" MacBook Pro with a 24" monitor, as well as the standard ipad and iphone. I don't have a desktop anymore and I do any hacking or side projects on a slicehost image. I used to have a pretty large collection of random computers and desktops, but over the years I have really cut back on hardware, it just doesn't seem necessary anymore. Like a lot of people here I also enjoy photography. I have a digital rebel xsi and a someone worse for wear 4x5 view camera, neither of which I use as much as I would like.

And what software?

I use TextMate (and sometimes vim) for coding. My development browser is mostly Safari, with Firefox/firebox for some things. I use Charles proxy a lot for HTTP debugging. VMWare fusion is indispensable. I have a portable hard drive with a little collection of VM images to test all the various browsers out there. I am also completely dependent on Evernote. I use it for everything. Frequently I remember something I need to do or figure out a problem while lying in bed, so I just grab my ipad, write down my notes and they are on my laptop when I need them.

What would be your dream setup?

Who are you, and what do you?

Ahoy! I’m Matthew Rothenberg. Most people here call me mroth. I’m the head of product strategy / management at Flickr, which means I’m ultimately responsible for figuring out what Flickr needs to build and why. I work with all of our teams during the development of Flickr features to help make sure they meet our goals and vision, as well as do boring manager-y things. In reality, that means I spend most of my life in meetings (formal or informal). In a broader sense, I’m responsible for trying to make Flickr the world’s best place to share your photos with the people who matter to you.

What hardware are you using?

I have a 15” MacBook Pro with 4GB of RAM (which I insist on still calling a PowerBook). I have a 24” Dell display both at work and at home, but I almost never bother to plug into them anymore. Mobile, oy--I have way too many phones. My primary phones that I carry every day are a BlackBerry 9650 (writing email, making phone calls I want to stay connected for more than 60 seconds), and an iPhone 4 (browsing the web and my primary camera). I also have a Palm Pre (used to be my main phone, now I use it mostly for development), HTC Evo 4G (development and testing) and a Nokia N95-3 (which I use as the world’s most expensive alarm clock). With all those phones, I’m a heavy Google Voice user for bouncing calls back and forth. For running all my side projects and home backup, I have a homebuilt Linux (Ubuntu Server 10.04.1 LTS) server with an Atom chipset sitting in a MSI Wind enclosure, and a Seagate “Green” hard drive that spins down to lower speeds to conserve power. My loft’s stereo is wired up for network play via AirTunes and Airport Express, but unfortunately the software for it is so terrible (no streaming from iOS) it doesn’t get much use currently -- I’m hoping AirPlay in iOS 4.2 will finally resolve this. More successfully I do stream video via the 802.11n network from the Linux server to my Sony PS3 which is hooked up to the television (Sony 40” flatscreen of some sort, if anyone actually cares). The most transformative tech hardware I’ve purchased in years has been the Kindle (2nd gen), which completely changed my relationship with books. It perfectly embodies the design principle of doing one thing exceptionally well, and has resulted in me reading approximately 350% more books since I purchased it. Finally, and most importantly, the best productivity-enhancing hardware I own is a La Marzocco GS/3 espresso machine.

And what software?

I live in email -- I currently use Mail.app for its live search folders but I’m not entirely satisfied with it. There is a definite market for a real “power user” email app that hasn’t quite been filled yet. Skitch is another “couldn’t live without it” application for me, since I spend a lot of time reviewing not-quite-ready-for-primetime features, it lets me super quickly capture and annotate notes for the team. I’ve been a UNIX nerd for a long long time, so I spend a lot of time in Terminal.app, and sadly even do most of my basic navigation file system operations in there. For actually hacking on code, I use TextMate. Most of my side projects are coded in Python, but lately I’ve been dabbling in Ruby/Rails. Github completely changed the way I hack on side projects. If you haven’t read Anil Dash’s recent blog post on it, you should. If you ever develop software for fun and you aren’t using it, you’re sorely missing out. I occasionally have to build presentations for my job, so when I do, I insist on using Keynote.

What would be your dream setup?

Who are you, and what do you?

Hello, I'm Chris Martin, one of the resident nerds here at Flickr who keep everything running smoothly. I'm from Atlanta, Georgia but I don't have a southern accent and this seems to disappoint people pretty regularly, I apologize for that. My focus at Flickr is on mobile engineering and making sure people can get their lovely photos out of their pockets and on to Flickr; and vice versa. In real life I'm a professional travel writer/photographer. I've never written anything and my photos are sub-par, but I have to justify traveling around the world somehow.

What hardware are you using?

At work I use a 15" MacBook Pro hooked up to two 24" Dell displays, one for code and one for browser windows etc., they are both quite useful for creating a barrier to hide behind during the regular FlickrHQ foam dart wars. I also have a variety of mobile devices that we use to test our own code, 3rd party integrations, and API apps. My Flickr Moleskine and stack of post-it notes are also integral parts of my work hardware setup. At home I have a pretty assorted collection of hardware, the most interesting of which is probably a 450MHz G4 Cube that sits quietly on my desk and still does a great job as a file server. Rounding out my fanboy worthy collection of Apple products are a unibody 15" MacBook Pro, an iPad, and an iPhone 4 that replaced my 1st gen iPhone which I've used for the past 3 years (and still do for testing). I have two "hackintosh" machines; one desktop with a couple terabyte drives hooked up to my Samsung TV as a media center, and a 12" ASUS 1201n which I find myself using more than my MacBook Pro because I like it's small size when I'm out, and I hook up to a 24" display when I'm at my desk at home. In my "retired" collection are a linux PC that used to be a MythTV media center, an original eepc 701 that served me well on a 5 month backpacking trip, an ancient original iMac, and an assorted collection of iPods. I have a problem saying goodbye. It's probably appropriate to include camera gear as well... I have a Canon 350D which usually wears a 50mm f1.4 lens; it's starting to show it's age but has seen and faithfully captured many great trips over the years, a Kodak M1033 which isn't the greatest camera in the world but it's built like a tank and was the least obscenely expensive thing I could find in India after leaving a Canon on a train, and a Panasonic Lumix ZS3 that takes excellent video. However, I still take most of my photos with my iPhone.

And what software?

Obviously I'm an OSX guy, ever since I moved from Gentoo linux to a powerbook in college. I love the polish of the Mac GUI, and the raw power of Unix that's just below the pretty surface. I use the assortment of standard apps (Safari, Mail.app, iCal, and address book) as my primary browser, mail, and calendar applications. I've moved around a bunch in the past in this area, but the apple tools have finally started playing nicely with the google suite of services and they work well enough to keep me happy and in sync with my iPhone. For development I primarily use TextMate (web-dev) and xCode (Cocoa), with the occasional trip over to Coda when I need to quickly touch something I'm doing any front end development on. I don't have any special tools I use for source control or file transfer, that's all svn, git, scp, etc. in terminal. A few other apps like Linkinus, Adium, and Echofon live in my dock all the time, and of course I pull out photoshop when I need to mock something up. As for utilities, I absolutely can't live without Visor, a SIMBL plugin that makes my terminal appear on command from the top of the screen, teleport is a nifty tool for sharing one keyboard and mouse between multiple computers, DeskLickr keeps my desktop beautiful with photos from Flickr, and MenuMeters gives me a quick view of how the computer is doing from the menu bar. On my phone, I use Echofon for twitter, Reeder is my absolute favorite RSS reader of all time (I wish there were a desktop version), Instapaper, AutoStitch to make panorama photos, iTimelapse for time-lapse video, and of course m.flickr.com and the Flickr app for browsing Flickr.

What would be your dream setup?

Who are you, and what do you?

I am Timoni Grone, former-Nebraska-farmgirl-now-Californian-city-dweller, and I design web interfaces. I'm currently a UI/visual designer at Flickr, which is a totally rad gig.

What hardware are you using?

At work, I have a 2.15Ghz Core 2 Duo 15" MBP, a 24" external monitor, the short Mac keyboard and a Mighty Mouse. At home, I have a 2.67GHz 15" MBP and a Magic Mouse. I forget how much RAM I have, but both computers are fast enough. I also have an old Mac Mini working as a media center (hooked up to my absurdly ginormous television), a 2TB Lacie Quadra for backups, an Airport for network stuff, a 16GB wifi iPad and an iPhone 4. I briefly had a Wacom Cintiq 12WX earlier this year. I excitedly used it for a few weeks, then left it untouched for months, and finally sold it. It was too much of a pain to switch back and forth between the tablet and my laptop.

And what software?

For web dev, I like TextMate, though I was on a strict Coda diet for a while when it made sense to have seamless FTP integration. If I'm using TextMate, I use Transmit. Current browser of choice is Chrome, cause it's super-fast, though I prefer Firebug to Web Inspector for development. Day to day, I use Photoshop and Illustrator all the time. I've been trying to switch to comping in Fireworks for a while, but for some reason it was never installed on my work computer, so I default back to Photoshop. The way it handles smart objects is a pain, but Save For Web in Illustrator can be effing tricky sometimes (also, the way Illustrator renders text below a certain size makes my eyes hurt). I have Skitch, the Last.fm scrobble app, and Dropbox running at all times on both of my MBPs. I use Dropbox to sync my work folder and one of my iTunes libraries. Works like a charm and it's only ten bucks a month. For print work, I use InDesign. For small things like bills & letterhead, I use TextEdit. I try to avoid halfway-done word processors like Pages and Word; the way they handle style sheets drives me batty. I just started using Notational Velocity for my to-do lists (thanks Daniel!). I use Google Calendar for scheduling, and have SMS reminders sent to my phone (a total lifesaver). When I want to write, I use Ommwriter, the nicest little text editor out there. On my iPad, I most often use NetNewsWire, Instapaper, the Kindle app, and Autodesk's Sketchbook Pro.

What would be your dream setup?

This is the first in a new series of interviews with the Flickr staff, asking what tools (hardware, software) they use to get the job done. Our first volunteer is Nolan Caudill, PHP engineer and wearer of beards.

Who are you, and what do you?

I'm Nolan Caudill, a software engineer at Flickr (obviously). I'm a recent addition to the company and a new transplant to San Francisco. At Flickr, I work on i18n/l10n, though I like to keep a variety of projects on my plate to keep things interesting. I'm originally from Wilkes County, North Carolina which is the moonshine capital of the world and the birthplace of NASCAR (and those two facts are indeed related). I moved here from the Research Triangle in North Carolina, going to school in Chapel Hill and residing in Durham. My wife and our cat moved here in May and are both loving this chilly, foggy city with good beer and food on every corner.

What hardware are you using?

I'm currently typing this on my work-issued MacBook Pro hooked up to a 24" HP monitor, fairly par for the course at Flickr. This is the first Mac I've worked on in several years and I'm a big fan. The consistency of all the software is nice to use and things just seem to work. At home, I use a Lenovo Thinkpad R61i with Ubuntu 10.04 installed. I've ran Ubuntu since version 4.10 and really enjoy being able to install about any developer tool under the sun with a one-line command. It's hard to beat the big Debian/Ubuntu repositories. This is also about my third or fourth Thinkpad laptop. They are built like tanks and are nicely supported out of the box by most Linux distributions. I use a iPhone 3G and it gets along just fine.

And what software?

As far as software goes, my programming life revolves around programming languages, browsers, and text editors. At work, I do almost everything in PHP and at home I'm always playing with new languages. I'm currently in the process of learning Haskell and learning how compilers work so I can one day build my own toy language. I've also done professional work in Python and that is still my favorite multi-purpose language where I can just get stuff done. My main browser at work is whatever the stable version of Firefox is. Like most web developers, I use Firefox for extensions like Firebug, Web Developer, and Y!Slow. I use Chrome at home, mainly just because I don't need a lot of bells and whistles when I'm browsing and with a smaller laptop screen, the minimal chrome really frees up some real estate. At work, I use TextMate to write code. This was a big departure from what I normally used. I've always used vim and have those keyboard bindings hard-wired into my muscle memory. I don't think TextMate is as powerful or as quick to use as vim (at least for me), but that's not necessarily a bad thing. TextMate shines at its intended purpose, which is to just get some code out. The project navigation and function browse is really nice and having it integrated in the Mac style is a plus. I still use vim for my off-hours hacking projects though. Other random pieces of software include: iTunes, Adium for both IM and IRC, Thunderbird for email, and Tweetie for keeping up with Twitter.

What would be your dream setup?

We’ve seen great uptake by library developers on supporting the new Stats APIs. Now we’d love to see what you’re building with those APIs.

PHPFlickr, Flickr.Net, flickrapi.py, CFlickr, and flickcurl all have let us know they support the new APIs.

Poking around the App Garden I found flickrstats which provides a very interactive, and lovely timeline based visualization. (in the style of Google Finance). Are there other great examples? Let us know by adding them to the App Garden or telling us about them in the API Group.

June 1st.

And a reminder.

Historical stats data is only available until June 1st. After June 1st only the last 28 days of data will be available. If you want to archive that data we have the stats download page, and a new utility method flickr.stats.getCSVFiles (because I was too lazy to manually download those files, see programmer virtues).

Ever since Peter Z from Percona showed us the awesome that is the mk-query-digest tool, we've been using it quite a bit to help identify costly queries. It analyzes the MySQL slow query log and compiles a detailed report which includes tons of useful data - how many times was a particular query called? how much time did MySQL spend executing a particular query during the given timeframe? how long did the query take to execute on average? what about the worst case? The report is output as a long long text file that you can page through and look at all the queries. All the data you need is there, but it's a bit difficult to get to, especially if you want to compare multiple reports or if you want to aggregate over a period of time longer than the interval at which you run the reports (we've been running it every 15 minutes on one of our shards and are in the process of adding it to all servers).

Enter ishmael

Photo from raphie.

We love galleries. After all, without galleries how would you find your giant sea bugs?

This post is to quickly announce we’ve added galleries to the API.

A Rose+GUID by Any Other Name ….

Galleries in the API use “compound-ids”. Like tags. An example gallery compound id might look like 9634-72157621980433950. Unlike photos you can’t simply grab the last number off a gallery url and stick it into the API. Yeah, I’m not thrilled about it either, but there are good (read boring) reasons why it works that way.

So when an API method says it takes a gallery_id, we’re talking about the compound-id.

You can however use the flickr.urls.lookupGallery method to go from gallery url to gallery_id. Pass the method the URL for the gallery, and we’ll give you back the gallery info blob.

You can also get gallery IDs from flickr.galleries.getList and flickr.galleries.getInfo.

Anatomy of a Gallery

Behold, a gallery info blob:

<gallery id="6065-72157617483228192" url="http://www.flickr.com/photos/straup/galleries/72157617483228192" owner="35034348999@N01" primary_photo_id="292882708" primary_photo_server="112" primary_photo_farm="1" primary_photo_secret="7f29861bc4" date_create="1241028772" date_update="1270111667" count_photos="17" count_videos="0" > <title>Cat Pictures I've Sent To Kevin Collins</title> <description>dive dive dive</description> </gallery>

The primary_photo_* attributes refer to the “cover photo” for the gallery. The owner is the Flickr user_id (aka NSID) of the member who created the gallery. The id is that compound-id we talked about.

Lists of Galleries

You can fetch all of a member’s galleries using flickr.galleries.getList, sorted from newest to oldest, returning a list of gallery info blobs.

Or you can fetch all the galleries a given photo is in with flickr.galleries.getListForPhoto.

A Bag of Photos

Perhaps most interesting, flickr.galleries.getPhotos will return a list of all the photos for a given gallery. It’s a standard photo response, with a twist.

<photos page="1" pages="1" perpage="500" total="15"> <photo id="2935475111" owner="8147452@N05" secret="e20746148b" server="3068" farm="4" title="Day off from the Death Star." ispublic="1" isfriend="0" isfamily="0" is_primary="1" has_comment="1"> <comment>best cat picture ever!</comment> </photo> <photo id="3078977730" owner="68779755@N00" secret="dba9d8105e" server="3229" farm="4" title=""We could stuff it with Kleenex..."" ispublic="1" isfriend="0" isfamily="0" is_primary="0" has_comment="0" /> <photo id="3212123792" owner="10983978@N03" secret="4231501383" server="3391" farm="4" title="1-19-09: Some People Just Don't Get It" ispublic="1" isfriend="0" isfamily="0" is_primary="0" has_comment="0" /> .... </photos>

In addition to standard photo response attributes, there is also a has_comment attribute which signals whether the gallery creator added a comment about why she included the photo, and whether the child comment element is present. Also is_primary, when set to 1, indicates this is the gallery’s “cover photo”.

CRUD

flickr.galleries.create creates a gallery, with a title, description, and optional primary photo, and will return a gallery element with the compound-id and the URL of the gallery.

<gallery id="50736-72157623680420409" url="http://www.flickr.com/photos/kellan/galleries/72157623680420409" />

flickr.galleries.editMeta is simply for updating the title and description. flickr.galleries.editPhoto confusingly doesn’t edit a photo, but rather the comment about a photo in a gallery.

Of course the money is all in flickr.galleries.addPhoto which allows you to actually build a gallery of photos.

Nota bene: Remember only public-safe can be added to galleries.

The Curated Life

We’ve also added the ability to restrict searches to only photos in galleries, with the in_gallery argument to flickr.photos.search

photo by xgray

After collections, the most frequently requested addition to the Flickr API has been Stats. Today, alongside access to some archived older data, we're announcing lots of new API methods that give you access to all the stats data we're logging. First up is flickr.stats.getTotalViews. If we call this with no arguments, it will give you the all time view counts for a user's photos, photostream, sets and collections:

# ?method=flickr.stats.getTotalViews <stats> <total views="584957"/> <photos views="391325"/> <photostream views="175691"/> <sets views="17856"/> <collections views="85"/> </stats>

Sounds simple right? There's a couple of details that are worth highlighting:

Not all Flickr members have stats enabled. If you get back an error, then someone needs to visit www.flickr.com/photos/me/stats/ and press the big pink button.

"All time" is a surprisingly vague concept. We started recording "All-time" view counts for photos, photostreams and sets sometime back in 2006, and started recording collection counts on October 21, 2008.

Now we know how many views someone got on their account, you'll probably want to drill down into when the views happened, where those views came from, and what photos were popular.

When

Getting details on when the views happened is simple, we can pass a date to flickr.stats.getTotalViews, and get the counts for just that that day. The date can be a unix timestamp, or something like "2010-03-24", but it has to be in the last 28 days as we don't keep detailed data around longer than that.

What

Getting an idea of exactly what photos have been viewed is also simple - you can call flickr.stats.getPopularPhotos. This returns a standard photos response, with an extra <stats> element giving the view, comment and favorite counts. As with flickr.stats.getTotalViews, this can be called with a date if you'd like to narrow down your results to just one day.

Where

Lastly, if you want to know where the views came from, you can call flickr.stats.getPhotoDomains. This gives a list of all of the domains that sent traffic to the photo. If you want even more detail on a domain, flickr.stats.getPhotoReferrers will tell you exactly what pages we've seen as referrers. Both methods can be narrowed down by both date or photo id if you want to filter the results more.

But wait! There's more!