Grandata's blog

In this project in particular, we analyze the viability of using cell phone records to lower the cost of urban planning, in particular, to find out how people travel in a particular city (in this case, Buenos Aires, our hometown).

We use cell phone data to make an estimation of the distribution of the population of the city and then compare it with more standard methods, such as urban polling (e.g. origin-destination surveys). These methods have a lower reach, as the number of people that can be interviewed at a reasonable cost is in the order of ten thousands (in the case of Buenos Aires). Moreover, these polling methods are held once in a decade in the cities that carry them on more frequently.

On the other hand, new methods based on cell phone data, although less profound in relation to the kind of data they provide, can be used (if they are proved reliable) to keep a real-time track of the flow of people among the different parts of the city. The objective, then, is to effectively prove them reliable enough, and then to explore the new possibilities that this methodology offers.

We have a dataset with approx. 2,1 million mobile phone users (1000 times the number of people in the Buenos Aires polls), and after filtering part of them, we keep 600K “valid” users. We consider that a user is valid if it provides us enough information about its mobility pattern (that is, the places where she/he has been). Here we can see the distribution of people among the communes, and their geographic disposition:

We listed all the calls of every user in a given timespan, in our case 5 months, and built with that information the “appearance map” for every person in our dataset. 

We differentiated four types of days:

Weekday (Monday to Thursday)

Friday

Saturday

Sunday

and four periods of the day:

Morning (5am to 11am)

Afternoon (11am to 3pm)

Evening (3pm to 8pm)

Night (8pm to 5am of the next day)

This division generates 16 “time cells”. Later we mapped each call to a commune, and aggregated the calls in each of them, grouping by commune, and normalized them to get, for each time cell, a probability distribution of that user over the communes.

We analyzed the user’s probability maps and kept those who appeared at least once in each time cell (so the total quantity of people seen in each time cell remains constant). After that filtering, we determined the home of the person (based on the predominant value, at night, of the location of the call records) and using that information we extrapolated our sample to the total of the population of the city (using each commune’s population, given by an official census, and the number of users of each commune according to our database after filtering). The sum of these values, over all the users, is the final estimate of people in each commune, for each time cell.

The first experiment was to compare with the urban poll’s data to verify the reliability of the method. In the following graph, we show the comparison for two very different kind of communes: commune 1 (there are lots of offices), and commune 14 (there are houses, and lots of bars):

   We observed that our method’s estimations and the poll’s are similar, which validates the reliability of our method. So the next step is to wonder what happens when we observe the weekends (there aren’t urban polls for non-working days). 

Here’s a graphic that shows the quantity of people for each time cell, for commune 14:

We can see that this figure makes sense, given that this commune is mostly known for its nightlife, and density of bars and clubs.

Last but not least, we have plotted the distribution of the origin of the people for a typical working day, for each neighborhood.

Here we can see some patterns. The dark diagonal shows that in most neighborhoods, a big percentage of the people there during the day are inhabitants of the neighborhood. Also, we can see more exchange of people among communes 12, 13, 14 and 15, so there’s many people moving inside that cluster of communes. Commune 1 is lighter in the diagonal than the others, meaning that it isn’t a residential area (the interesting thing is that this matches our insight of the city).

We think that this method to estimate the population can be very useful as a complement of the traditional methods, as it allows to have a real time analysis at a considerably lower cost. To continue working on this we will consider other regions of interest, or try to have more level of detail in the mapping, but there are lots of possible expansions of this methodology.

During the workshop we had very interesting presentations. Artur Ziviani (from LNCC, Brazil) talked about a formal framework to work with time-varying multilayer networks. Marton Karsai (from ENS Lyon) gave a great presentation on spatiotemporal correlations and information spreading in mobile phone communication networks. Eduardo Mucelli (from Ecole Polytechnique and INRIA) spoke about measurement-driven mobile data traffic modeling in a large metropolitan area. We also had presentations from Werner Creixel (USM, Valparaiso, Chile), Hakima Chaouchi (Telecom Sud Paris) and Anelise Munaretto (UTFPR, Brazil). I presented with Jorge Brea our work on the inference of users demographic attributes (such as gender and age), based on their mobile phone usage patterns, and the topology of their social network.

After the workshop, I spent more time working with the group at INRIA Saclay, in the campus of the Ecole Polytechnique, and with the group at ENS Lyon. These were very productive days, and we are advancing in a wide range of projects: 

detecting socio-cultural events,

analyzing and modeling the demand of web traffic of mobile users,

understanding correlations between different layers of human behavior (e.g. their spending behavior and their mobile phone communications), using the fact that we have a dynamic view of both layers,

summarize the geographic mobility information in aggregated variables, which can be used to refine other classifications (e.g. the age and gender classification).

Also, in Lyon, I got the chance to see the Lego Turing machine made by Yannick Leo and others (link).

Datos Abiertos como operador de apertura

El movimiento Datos Abiertos es mucho más que datos que ya son, o deberían ser, abiertos. Es tácticamente astuto comenzar por aquellos datos, como los generados por los Gobiernos, que por su naturaleza ya son abiertos. Pero es estratégicamente necesario reconocer (y luego planear y luego actuar) que el mayor potencial está en transformarse en la fuerza brutal y creativa que empuja a los datos a abrirse. Me faltan referencias, pero estoy seguro que lo que actualmente consideramos Datos Abiertos son una proporción insignificante comparado con los datos cerrados. Allí se esconden 3 billones de oportunidades.

¿Qué mundo tendríamos si todos los datos fueran abiertos?

Me resulta productivo pensar los extremos. Podríamos decir que #DatosAbiertos surge como respuesta a un mundo donde los datos son/están cerrados. Es el extremo opuesto que me interesa: un mundo donde todos los datos son abiertos.

De más está decir que no tengo idea qué mundo sería, pero sí que esta pregunta nos confronta con ciertos problemas fundamentales que siento debemos encarar entre todos.

Usando la analogía del –ya démodé– laissez-faire, que pretendió minimizar la intrusión del gobierno en la esfera económica. Postular ese límite nos ayudó a pensar qué mundo queríamos y abonó un debate que marcó la historia.

En este mundo imaginario donde todos los datos son abiertos un fenómeno interesante es que al "liberarse" los datos, la escasez asciende un nivel en la cadena de valor. Ahora lo escaso no es el dato (o mejor dicho, su acceso), sino la capacidad de procesarlo/darle sentido: una mezcla no evidente entre algoritmos, poder de cómputo, capacidad de almacenamiento y materia gris.

Este nuevo escenario me resonó con la historia que contó Santi en el panel, que los escribas en la edad media no sabían leer; y también con la idea de Mckenzie Wark que el capitalismo actual tiene más que ver con la posesión de los medios de mediación que con los de producción (leer).

Este panel fue moderado por Joel Gurin, con participación de Jeff Kaplan de Socrata, Moís Cherem de Enova, Alex Cantú de SkyAlert, además de la mía.

Como algunos sabrán, en Grandata trabajamos principalmente con “datos cerrados” (voy a remitir a la idea intuitiva y escapar a una definición que me excede). Es por esto que en el panel expuse los tres tipos de interacciones que encuentro entre Datos Abiertos y datos cerrados:

Los Datos Abiertos como recurso complementario a datos cerrados. 

Usar datos públicamente disponibles para enriquecer análisis basados en datos cerrados. Existen muchísimos ejemplos, pero uno divertido de Grandata es cómo usamos la información (pública) del cronograma futbolístico de Argentina para mejorar la predicción la localización de una persona usando la información (cerrada) de movilidad de teléfonos celulares. Leer la cobertura de MIT Technology Review.

Los datos cerrados como generadores de Datos Abiertos.

Esto es cuando usamos datos cerrados como insumo de procesos cuyo resultado son Datos Abiertos. Un ejemplo es nuestra colaboración con MIT Humnet y el Banco Mundial, la idea fue usar datos cerrados para generar información que gobiernos puedan hacer pública, sin revelar los datos (cerrados) subyacentes.

Abriendo los datos cerrados.

La gran pregunta es por qué. Por qué Google o Goldman Sachs o Walmart abrirían sus datos. No sé la respuesta, pero creo que existe gran potencial en abrir los datos para el bien social (open data for social good). Ya hay ejemplos: el gran trabajo de Nicolás de Cordes en Data for Development de Orange, la iniciativa de Big Data for Development de LIRNE en APAC, o la utilización de datos celulares para combatir la Malaria.

Estos ejemplos son importantes por que demuestran que abrir datos tiene impacto concreto y medible para el bien social.