#pysal

I wrote a little guide for my fellow library devs about how to use soft dependencies effectively. This moves in tandem with some work on enabling PySAL to have a consistent user experience while leveraging the newest & best libraries for scientific computation. Since most of the work I’ve been doing involves using soft dependencies with the library, it’s important to get this right. 

I've had to take a break from the spatial hierarchical linear modeling kick I've been on recently to get back to some GSOC work.

Today, I had my weekly call with my mentors.

On today's call, my mentors and I discussed a few things.

Testing & Merging of project code

Since many of the improvements I'm making to the library are module-by-module, I was advised to submit PRs when a logical unit of contribution is ready to ship. As I've already been trying to use Test-Driven Development principles for my project, writing a test of what I want the API changes to look like then writing to that specification, this is relatively simple: the module is ready to submit when the spec tests pass. So, now that much of the initial foray into the labelled array API is done, I can beging to connect the tests & submit PRs where possible.

Appropriate Targets for Labelled Array interfaces

We also discussed how to extend the labelled array interface to other submodules. For example, I have a good idea how a consistent labelled array interface could look for Map Classifiers in the exploratory spatial data analysis module. Other elements of that module should also be relatively straightfoward to implement on labelled arrays, since all that's generally needed is input interception: dataframe+column name needs to get correctly parsed into numpy vectors. This is quite simple. The spatial regression module also seems like a relatively straightforward place to add a labelled array interface, using a similar strategy to what I've already been doing in pysal.weights. Defining a from_formula classmethod for spatial regression modules would allow for specification of regressions using patsy & pandas dataframes.

But, in other parts of the library, like region or spatial_dynamics, it's less clear as to what the labelled array interface should look like, so I'll have to gain some perspective there.

Remaining Confusion in weights construction

I'm running into some minor confusion because I'm trying to make a call like from_dataframe(df, idVariable='FIPS') equivalent to from_shapefile(path, idVariable='FIPS'), and can't figure out when PySAL considers things ordered vs. unordered.

For background, a spatial weights object in PySAL encodes the spatial information in a geographic dataset, allowing estimation routines for various spatial statistics or spatial models. In doing this, it relates each observation to every other observation, using information about the spatial relationship between observations. In our library, these are used all over the place.

But, in building a new, abstract interface to the weights constructors, I got quite confused. Particularly, I was expecting to be able to write a pair of classmethods, say, Rook.from_shapefile() and Rook.from_dataframe(), that have similar signatures and generate similar results. Something like from_dataframe(df, idVariable='FIPS') being equivalent to from_shapefile(path, idVariable='FIPS'). Unfortunately, it's somewhat confusing to figure out how to make this work correctly, without making the API incoonsistent. This is because PySAL handles ids in weights objects across its various weights construction functions and classes in different ways. I think, overall, we expose four different variables or flags at different points in the API that deal with how observations are indexed in a spatial weights object:

ids - ostensibly, a list of the ids to use corresponding to the input data, considered in almost every weighting function.

idVariable - a column name to get ids from when constructing weights from file used in existing from_shapefile functions to generate ids.

id_order- a list of indices used to re-index the names contained in ids in an arbitrary order, impossible to set from from_shapefile functions but used in the weights class's __init__

id_order_set- a boolean property of the weights object denoting whether id_order has been explicitly set.

To me, this is rather confusing, despite some conversation trying to flesh this out.

First, all lists in python are ordered. So, when a user passes a list of ids in as ids, its confusing that the order of this list is silently ignored. Second, when we construct weights from shapefiles using an idVariable, the resulting weights object has some peculiar properties: the id_order is set to the file read order, but the id_order_set flag is always False. This is confusing for a few reasons. First, shapefiles & dbf files are implicitly ordered, so a column in the dbf should correspond exactly to the order in which shapes are read, barring data corruption. So, if I use a column of the dataframe to index the shapefile, this should be considered ordered. Second, our docstring below seems to imply that either id_order_set is False and id_order defaults to lexicographic ordering, or id_order_set is Trueand id_order has special structure:

id_order : list An ordered list of ids, defines the order of observations when iterating over W if not set, lexicographical ordering is used to iterate and the id_order_set property will return False. This can be set after creation by setting the 'id_order' property.

But, one can easily generate an example where id_order is not lex ordered and id_order_set is False:

import pysal as ps

Qref = ps.queen_from_shapefile(ps.examples.get_path('south.shp'), idVariable='FIPS')

Qref.id_order_set

False

Qref.id_order

[u'54029', u'54009', u'54069', u'54051', u'10003', ...]

Qref.id_order_set

False

NOTE: A demo of the relevant code I'm referring to for the new labelled array API in pysal.weights in this update is available in this notebook, and the actual code lives in a weights2 module in my gsoc feature branch.

I've decided to target our weights module to prototype the labelled array interface. In general, we'll need extensions built into at least our exploratory spatial data analysis module, esda, our spatial regression module, spreg, and our spatial dynamics module, spatial_dynamics.

So far, I've focused on weights because it's so central to everything else that the library does. It also poses unique challenges to deisgn around, and I've already done a bit of work before GSOC in making a labelled array interface for it.

I've been building constructors that let us build spatial weights objects from the primitives defined in various other computational geometry packages. Fortunately, what's required is nothing more than building efficient type conversions &, where possible, relying on duck typing.

I'm somewhat concerned that just relying on duck typing may make this API more fragile than we'd like, so I'm trying to be eager about conversions to our native geometric objects, as long as the conversion is computationally cheap.

Altogether, this means that I've done quite a bit of redesigning of the weights module. But, in general, it still supports the same basic interaction style, but now can build weights from arbitrary iterables of shapes or PostGIS-style dataframes.

Wow, did not expect this to be as big of an issue as it has been.

Pysal removed the code for generating voronoi diagrams for the standard library. Not sure why.

Instead, on the mailing list, the project BDFL provides code for a diagram of randomized points.