The Electric Rice Cooker

This graphics package on the Hong Kong–Zhuhai–Macau Bridge is pretty much my swan song at the South China Morning Post! It was an exciting two years of learning and experimenting, fulfiling a childhood dream to work for Hong Kong's English newspaper. This project encompasses many of the new techniques and technology learned in the last few months, including processing Landsat satellite imagery, ai2html, ArchieML and Illustrator (!). It couldn't have been done without the generous contribution of my dear colleagues of all desks (like for any project done at the Post so far). For that I'm infinitely grateful.

Great maps and great video/photo interactive on the alarming melting of Greenland. I was curious how the zooming effect was executed. So I fired the Developer Tools. First, I went in with the element selector on the zooming maps. Turns out it's a canvas tag! Then, I stumbled a bit to find how the imagery was rendered, until I remembered to check my Network tab. So it turns out that there's a succession of images that get loaded to your browser, from a map of Greenland in its context to an image of the scientists' camp. The images even seem quite optimized / reduced in quality (to around 300 K each), probably to make sure it loads quickly when a reader scrolls to that part of the interactive feature. I guess that how each "frame" melts into the next is a play on the opacity of the image, but I didn't identify the code. Judging from the URL structure, there's probably different versions of the images for different screen sizes. It's very nicely done, as always, from NYT Graphics.

Great technology allows for great storytelling, which seems to be the motto for the always excellent NYT Graphics team. In their newest graphic, How ISIS Expands (Hannah Fairfield, Tim Wallace and Derek Watkins), the Times looks at strategies of ISIS' expansion, published on the day the ancient city of Palmyra fell. One of the great technologies underlying each project at the NYT: ai2html. Since March 5, the Illustrator script was made available to the public. What it does is convert AI files directly to HTML, which I've described last summer before actually seeing and using the script (some guy even went as far as creating his version of it, which of course doesn't handle all the options the NYT ai2html has). Again, you need to know how to use it, and for that is another challenge for every modern graphics desk in media organisations. When integrated well into your workflow, it aims to provide a tool that lets your designers design the way they always had using Illustrator and produce an output that is for the Web. It segways well into the other point that makes something like the ISIS piece so great. Inside the "ai2html" script (which reads like JavaScript?) is a part where some "NYT break points" turn up, circa line 668 of the script's version 0.54: It turns out that when you create multiple artboards in your Illustrator file, they will be generated with different HTML classes for the size they are optimized for. We leveraged that for our own graphics, like the third runway piece. You can even force the artboard to work for multiple break points, by adding the size it's meant for after the colon (see screenshot before). This is what it looks like when you squeeze it to a mobile screen size: The next "technology" in this piece: Swiper. Presenting a topic in the form of "slides" (on the piece's mobile version) is something that I find very interesting and would like to try out someday. We don't often get the chance to do it in our newsroom, where print is still king, but at least we know that it's possible to integrate elegantly. In this case, when you're on a mobile screen, first of all, you are given the narrow width artboard that was created on Illustrator via ai2htm. For instance, this is the PNG background of the first "slide": http://graphics8.nytimes.com/newsgraphics/2015/05/15/isis-next/db6b19f7946afe46e0f01b83b7acb5575eab2aeb/isis-next-isis-control-map-hed.jpg Then, the Swiper script encloses your graphic div as a slide, so that touch, click and mousewheel events trigger going to the previous or next slide. That's incredibly simple, and yet I'm a bit impressed by it. The next thing that I like a lot from NYT maps (other media like WaPo do it too on breaking news) is the addition of relief. I know it's been done on print for many years, but I wish we had the ability to have it in our map library. As a smaller newsroom, lightly mechanized one, we often still trace our maps by hand. I'm going on a segway, but last week, I helped my colleague colour this map of congressional districts (for Chinese investments in the US). I learned on deadline that QGIS is a hassle to export from (and you can't export in vectorial formats that are stylized). The solution wasn't perfect and originally in the wrong projection. But print media approximations made things look smooth in the end (but I don't really like the feeling). So, again, when you have the tech ready to be used, your team can focus on doing journalism.

For the first time at SCMP, we published an original back page with an interactive graphic (online) adaptation on the same day. The topic was China's overseas investments, visualising the magnitude of all deals, mergers and acquisitions, made by Chinese companies, many of which are state-owned entities. So, my colleague Alberto finished the paper back page graphic (full page broadsheet infographic) sometime last week, ready for publication this week. On Sunday night, I did a simple layout of the semi-circles, a brilliant idea, using D3. Over the course of Monday, I worked on layout and the legend. Then finally on Tuesday, as the print version came out, I went over the graphic with Alberto to get his comments and make sure he was entirely good with the end product. We spent this memorable 15-minute back and forth this afternoon, deciding what we'd do with those annotations (on the Rio Tinto and Nexen deals), finally settling on making the notes slimmer (because by mistake, I forgot to wrap the div's width attribute value in a function, it took a minimal width) and give them a border and white background, tooltip-style. The rest of the text was handled using ArchieML, although there was hardly any editing to do, and I was the only user. The graphic is responsive (it re-draws when hitting new break points), but doesn't retain all of its functionnalities on small screens. I'm pretty happy about the two-day turnover, but it could've been done quicker, if it weren't of endless fiddling with the legend and notes.

The big highlight of the last month was certainly going to NICAR, which was held in Atlanta on March 5-8. Between the runs to the liquor store and BBQ joints, I met a lot of fellow Canadians, and some of the journo-programmers whose work I've been admiring from afar. Here is a list of what I remember the most: - ai2html: The excellent Illustrator script was released to the public on the first day of NICAR. It has been one of the key tools made by the NYT graphics team that allows to convert Illustrator files into straight up HTML. An Illustrator artwork is decomposed into absolutely positioned text and a background for the remainder of the graphic. Tried it out immediately after returning to Hong Kong and it works pretty well. Now, inserting that into your website, whether it's responsive or not, that's something we've yet to figure out. - Space journalism: This past year has seen a bunch of really cool project making good use of satellite imagery for storytelling, such as ProPublica's Losing Ground and NYT's A Rogue State Along Two Rivers. For some reason, I never thought it could be (partially) made with publicly available information that you could put together with widely used tools (GDAL or Photoshop). There was a session on the first day and again another session on the topic during the lightning talks. - Everyone's just trying: The great thing with these conferences is that you get to measure your level of skill, etc., with others in your industry. And in fact, success is multi-factored and it's worth just trying things out. Maybe something will stick? It convinced me to write down what my wish list would be if I were to hire for my position.

It's a step in the right direction. That's my assessment of the raw data released by the Hong Kong Financial Secretary's Office, along with the Financial Services and The Treasury Bureau at the same time as the budget speech on February 25, 2015. The data is on a page that's parallel to the one where the PDFs have been traditionally released for the last 15 years. I for one welcome the initiative. It has had a practical impact on the work we've been doing at the Post. Instead of having to update our dataset with scripts, I copied it from the CSV files. We met representatives of the government a couple of weeks before the release of the budget. The press secretary happened to be a computer science major, which certainly gave him a sensibility to the technical aspects that programmers seek when looking for open data (the Financial Secretary himself is a MIT grad -- at the school of architecture and planning). This release follows months of emailing (with a long gap in the spring and summer) with our friends of Open Data Hong Kong and the FSO. It shows it can be possible to encourage change in the government practices. As for more extensive use of the data, I doubt we will have a good case until data gets released for past years, and actual details of some numbers that only the individual departments have also be known to the public. In the meanwhile, the PDFs already prove to be an infinite source of news, if you know how to navigate them. My colleague Alan Yu had a marvelous find just by checking out expenditures of the police force (they bought three super-expensive crowd-control vehicles).

Edit (2015-02-22): I've added the scripts on GitHub that are working for party contributors, and partially working for riding associations contributors. I'm starting the scrape for contribution reports (postal code, etc.), but it's extremely slow (6 pages per minute). It's probably because the machine I'm using is in Asia. If you have a bunch of machines on a good connection, let me know! Since 2009, I've been on the case of the Elections Canada contributions database, where all donations to political parties are "accessible" to the public. In the summer of 2013, the site was revamped and made considerably harder to scrape. You could always get the basic infos, like the name of the contributor, how much they gave, and even a unique identifier "client id" for each person. But for the rest, the postal code, municipality and province, that really allows you to do more serious crossing with other databases (or for maps), you couldn't retrive it in a simple way like before, when that personal information was found on popup pages, and a relatively simple URL without particular session check. So actually, I had given up at that point. But with a federal election coming up, and e-mails flying here and there about the election, I decided to put some time to poke around the site. When you scrape a site, the rule is always to understand how a human navigating the site goes around to get his information, what he triggers in terms of cookies, IDs, etc. My tools of predilection is just Chrome Developer Tools. Then, I use the handy "Copy as cURL", by right-clicking requests in the "Network" section, while navigating the site. In this case, I went to the donations to the parties, the largest of all the ways money enters the coffers of political parties (money may be sent through riding associations and candidates, among other ways). I noticed that the URL at the top changes, but if you just paste it in an incognito window (good way to test whether a page is also dependent of state), you are sent back to the homepage. E.g.: http://www.elections.ca/WPAPPS/WPF/EN/PP/ContributionReport?act=C2&returntype=1&option=4&queryid=b2d4d650d0304b2ebf46a3dff36d70a6&period=0&fromperiod=0&toperiod=0&exactMatch=False&contribrange=-1&contribclass=1%2C%206%2C%2012&selectallcontribclasses=True&addrclientid=24447&addrname=Jean%2BA.%2BCorsi&displayaddress=True&reportPage=1&totalReportPages=6127&ln_no=5318339&setfocusid=addresslink53183391 sends you back to this page http://www.elections.ca/WPAPPS/WPF/EN/PP/ So, quickly I identified the cookie as being crucial to getting the right pages. After poking around a little, I found out that what was even more important was the queryid. The queryid is generated when you select the parties you are searching. Of course, I will want to select all parties at once. When you click the button, it is as if you registered with the webapp as to which entities (an entity represent one annual/quarterly report sent by parties or candidates to Elections Canada), and the webapp returns you a valid queryid that you can now use for your scrape. I usually take the cURL command from the Chrome Dev Tools, but try to reduce it as much as possible, taking parts out and keeping the minimal version that would still yield valid data to be returned. In the case of parties, to get a valid queryid, I found that you don't actually need a previous queryid (even if on the Web, you will have one), and can blank out most of the parameters of the post data, but you will require the cookie, and probably the basic search parameters in the query string (as well as returntype in the post data, as whether the contributions are verified or not, I think). When you issue the cURL, the page will redirect you to a page with the next search page in the form of: http://www.elections.ca/WPAPPS/WPF/EN/PP/SelectSearchOptions?act=C2&period=0&returntype=1&fromperiod=2007&toperiod=2014&queryid=884ebe6c192d489bb17f842c080f86f8 (This won't work in your browser, because you may not have a valid cookie or the queryid will be invalidated by the time you read this.) After that, you can go to the report page and try to download all the listing pages: http://www.elections.ca/WPAPPS/WPF/EN/PP/ContributionReport?act=C2&returntype=1&option=4&queryid=7f48da109479493aac937abbbded46f8&period=0&fromperiod=0&toperiod=0&exactMatch=False&contribrange=-1&contribclass=1%2C%206%2C%2012&selectallcontribclasses=True&addrclientid=25922&addrname=Jocelyn%2BT%2BAantjes&displayaddress=True&reportPage=1&totalReportPages=1207&ln_no=5816619&setfocusid=addresslink581661930 As of today, there are 6127 pages of 200 entries each for contributions to parties, to all parties, between 2007 and 2013 inclusively (details of 2014 aren't in yet). What we'll be interested to see are the quarterly reports for 2015 later in the year, as that is the most lucrative time for parties. The same strategy should be mirrored for other entry points for the money. With the data, I would check for irregularities, like signs of weird amounts, or simply try to draw insights on the patterns of money collection. It's also a massive, very massive database, with in excess of a million entries, surely hundreds of thousands of unique contributors. Finding something in there is daunting. But if a simple data source can be provided, then you're half-way there already.

China maps GIFs from our two most recent data stories at SCMP on air pollution in 2014 and ghost towns.

But I can tell you at least that we used Google Sheets as our CMS to arrange our circles in a rudimentary 2-D grid and to let our writers edit text directly in the people connections graphics.]

One of the regrets is that there weren't any other interesting close-up webcam on a crowd aside from the Hennessy & Yee Wo one. The one at Garden Rd & Queensway had some cool shots on September 28 as police was gathering, having a pep talk, sleeping on site, on the footbridge.

(That's from a Legco Hansard document from July 2, 2014 from page 156 / 16346 -- see database on the particular policy issue of ICH) I was sure bean curd sheet (腐竹), not just regular bean curd itself could've made the list if the cultural bearers returned the phone calls! It would be nice to have those tables in that PDF in a good machine-readable format, but it's probably not possible right now (I didn't verify beyond Google), even though Legco is known in the community as leaders in open government data. Well, to be continued...

The Big Four graphic was published during Wimbledon in early July and the design is thoroughly explained on this Tumblr. It could need a "novice layer", but I think it at least succeeds in telling the big picture of the Big Four's dominance of tennis, over the last 11 years. The 2014 US Open marks a turning point in the apparent end of this dominance, with the losses of both Federer and Djokovic in the semifinals. For the first time since 2005, neither of them or Nadal are in a Grand Slam men's singles final.

Archives! Diagrams! Interactive graphics! Videos! Our team at SCMP published a new multimedia extravaganza today.

It's work that we are very proud of publishing on SCMP.com this weekend. Over the course of the last few weeks (on and off), we've been working on the story of China's involvement in the first world war. My colleague Patrick Boehler wrote the story, gathered the extraordinary documents (photos, maps, illustrations -- even the poster of a movie made by Paramount Pictures), and produced the piece. It came out late Friday night and in the newspaper over the weekend. A back page infographic made by Alberto Lucas López also appeared both in Saturday's paper and online. Now, more about the techniques - First, there was skrollr, a pretty neat JavaScript library that helps create parallax effects that we like so much. We used that to a certain extent (for fading, mostly) in the Tiananmen piece. Skrollr.js has also been used by The Guardian (in 2012!), and also Sports Illustrated in their longforms such as this one on ice hockey. It was also used, beautifully so, by The New York Times in an illustrated feature on MMA fighter Charlie Rowan. - Then, to hold things together, we used Bootstrap.js, which is a framework for making responsive sites (leveraging the CSS "@media" query). It's the first time that I customize it using a CSS generator like Grunt. I winged it on the previous project. - The timeline at section 4 was obviously custom-made in D3.js. I wanted to make the maps in D3.js as well, but it would've been too time-consuming (so our fantastic art dept did them in AI). That's what brought me to wonder about tools to convert AI files straight to HTML. - Finally, as an attempt to improve performance on slow networks (although that shouldn't be a problem for most HK users), I called upon the Google tool/service Page Speed. That basically taught me a few things on pictures, number of colors in an image and colorspace. It's especially relevant to our project, since we have black and white pictures that we showed full screen. (If I didn't tell you, you probably would've guessed from the source code.)

That triggers a bunch of stuff: saving of the elements such as the iframe with the multimedia, nuke() to delete all the non-necessary elements, maximise() to expand the iframe to fill the screen. There you go! Nice trick to keep a single URL for sharing around and that is within your CMS' infrastructure and template. But sounds like many hoops to go through for just that.

Suzanne came back from the US and got me a copy of the Sunday New York Times of the week before. *More glorious pics of here!!* Otherwise known as the legacy medium of the NYT, it elicits a certain aura of mystery for someone who's never lived in the U.S. I'm not sure if I like paper. I just subscribed to the iNYT, because you get the online all-access with it and altogether it's cheaper than an all-access by itself (at 185 HKD/mo, it's about the same thing as the 20 USD/mo for the iPad subscription). The last newspaper I subscribed to was the Globe and Mail, and I think I stopped around 2002. For sure, the main interest (at least to me) of paper is the surface you're given for cool data visualisations. It's simply something that electronic devices still can't beat (until the portable projector becomes mainstream *g*).