Lines as Long as You Like

Today, inspired by a recent donation*, I finally got around to speeding up the display of long route lines (i.e., the line marking a route on the trip planner map). Previously, long route lines would take a long time to show up, and when a line was too long, it just wouldn’t show up at all, possibly crashing the user’s browser. Obviously not good.berryjam.ru

What happened was, I got notification of a donation and saw that the person making the donation lives in Beaverton. I tried getting a route from Beaverton to downtown Portland, but the route never came up because the line was too long. I felt bad that someone had made a donation but might not even be able to get directions over a relatively short distance, so I got to work on fixing the problem.rtisnab.ru

So, skipping over the technical aspects, I’ll just say that now it should be possible to get a route of just about any length. It still takes a while to generate a really long route, but once the route info gets back to the user’s browser, the line drawing is almost instantaneous. Even for short routes there’s a noticeable speedup.

This has been bugging me for a long time, so I’m glad it’s finally fixed. As usual, if you notice any problems, please let us know.

* Big thanks to Rebecca in Beaverton.

Scroll Wheel Zooming Added

A few days ago Google added scroll wheel zooming to their public maps API. Today, I enabled that feature in the trip planner. It was really simple, only requiring the addition of one line of JavaScript code.

This feature has been available on the official Google Maps site for a while, but it wasn’t available to third party developers until three days ago [1].

Another feature Google released recently is the ability to encode long lines for more efficient rendering. For long routes, this means the annoying “this script is taking a long time, do you want to continue popup” shouldn’t come up again.

We haven’t incorporated this into the trip planner yet but will some time in the next week or so (it’s a bit more complicated than adding scroll wheel zooming). I wrote some code that does the line encoding; now I just need to set up the back end to encode lines before sending them to the Web.

[1] Google Maps blog post about scroll wheel zooming

Upgraded to Python 2.5

I just got done installing Python 2.5 and converting the trip planner code to use it instead of 2.4. The only required change was pointing the “dispatch” file at the 2.5 executable. No other changes to the code were required to make it run on 2.5.dekor-okno.ru

A few little things also got cleaned up, and I updated to the latest versions of MySQLdb (1.2.2b3) and simplejson (1.5).

Hopefully nothing will blow up.

Problems with the Trip Planner in the Last Day or So

Our hosting provider moved all our files to a new server–without letting us know before hand that they were going to do so. Most things kept working. A few did not, as you may have noticed.focuz.ru

The reason some things stopped working is, the new server is running a newer version of the operating system and some of the software the trip planner depends on[1] had to be recompiled [reinstalled from source code].

Though I think the upgrade is a good thing overall, I hope they let us know about it before they do it next time!

[1] The MySQL bindings for Python, in case you really wanted to know. I also updated Python from 2.4.3 to 2.4.4, but I don’t know if that was strictly necessary.

Python Cartographic Library (PCL) — Installing Just the Spatial Package

So let’s say you need a spatial geometry library for Python. You could write your own; you could also use the PCL. The PCL includes some packages we don’t need, like one for MapServer rendering. I only installed the minimum needed to get the spatial package working (which I’ll talk about below).reteks.ru

[Note: “geometry” refers to points, lines, polygons and other geometric forms used to represent real-world objects. Examples: intersection (point), street (line), zip code boundary (polygon).]

I wrote a rudimentary geometry library for the trip planner that’s been working fine, but now I need to do some more “advanced” stuff related to using PostGIS and SQLAlchemy. In particular, I want to convert database values to Python objects and vice versa.

The first part (database to Python) is fairly easy and our current library already does that, but it’s convoluted in that it gets the database value as well-known text (WKT), parses that, and creates a Python object. From what I can tell, the PCL can go straight from well-known binary (WKB) geometry to Python objects.

The second part (Python to database) is harder because it involves converting a Python object to a binary geometry value. I don’t know anything about the binary geometry format and I don’t want to know, and it looks like with the PCL I don’t need to know.

I’m assuming PostGIS and PCL will get along together because they both rely on the same libraries, proj4 and geos. We’ll see.

The installation was fairly straightforward. The PCL includes five sub-packages. We had to install two of them, PCL-Referencing and PCL-Spatial. PCL-Referencing requires proj4. PCL-Spatial requires PCL-Referencing and geos >= 2.2.2. Something in there also requires the OGR library, which is included with GDAL.

The basic steps are, install proj, geos, and gdal, then install PCL-Referencing, and lastly install PCL-Spatial. On Ubuntu 6.06 (Dapper), here’s what I actually did:

Test Driven Development, PostgreSQL, SQLAlchemy

I just tried out some o’ that new-fangle “Test Driven Development” (TDD) I’ve been hearing about. Yeah, it’s good stuff.veroxybd.com

At the moment, I’m in the process of migrating GIS data from MySQL to PostgreSQL so we can take advantage of the PostGIS spatial extensions. I’ve also been making a bunch of related changes (AKA refactoring) in the “model”, separating things that never belonged together, and so forth.

I started out by rewriting the MySQL data import script. [Note: basically, the script pulls data out of a flat ESRI shapefile and normalizes it.] This wasn’t a complete, from-scratch rewrite–a lot of stuff I just copied over and tweaked a little bit. The biggest changes here were due to using SQLAlchemy instead of typing out SQL queries. I’ll just note that SQLAlchemy is “da bomb” and makes many things easier (once you get the hang of it).

That part was pretty straightforward, and low-level–I didn’t get into the ORM aspects of SQLAlchemy at all.

The next step was to modify the routine that creates adjacency matrices for routing. In the end, this was straightforward too. I ended up reusing some stuff from the new import script, which was cool. I refactored a lot during this process, adding some new modules and classes.

[Here’s where we get to the TDD aspect.]

So, I was sitting there (here really) thinking, “Hmmm… what now?” I drew some diagrams with the new classes and associations…. OK, that’s fun…. “Wait, I know. Run the unit tests!” Doy!

I started with the address normalization service, since the other services both depend on it. The test suite for this service isn’t as comprehensive as it probably should be (there are 19 tests), but it proved to very useful for shaking out a bunch of bugs in all that refactoring. The tests also helped keep me focused, and that aspect might be more important than the bug-squashing aspect (maybe).

Today, address normalization. Tomorrow, geocoding.

Foray Into Pylons (Python Web Framework)

For a while now I’ve been planning to start using one of the many available Python Web frameworks in the development of the trip planner. Over the course of the project we have built an ad hoc, application-specific, CGI framework (of sorts). Using a “real” framework will allow us to stop refining those wheels we reinvented and instead focus more on the specifics of our application.??? ??????? ??????? ?????? ??????.

Deciding which framework to use has been tough. In the past year or so it seems Web frameworks have become all the rage, with several interesting Python projects and all the Ruby on Rails hype. I had initially narrowed it down to two choices, having researched just about all the competitors: TurboGears and Django. I don’t remember Pylons in that first round of research, but maybe I just missed it. Once I noticed it, it was added to the list too.

So, the idea had been floating around for a while and I had created several test applications with various frameworks but still hadn’t gotten around to making a definite decision and switching over. Recently, I listened to a podcast interview of Guido van Rossum[1]. In it he mentioned that Django is his favorite Web framework. He didn’t say much about why except that he likes the way the project is being run. (“They really get open source.”)

Having heard that and having also narrowed the field to just Pylons versus Django, I decided, “OK, well I’m going to go ahead with Django then.” I ordered up a new Ubuntu VPS[2] to host the new version of the trip planner on and got to work. It was exciting to finally make a decision and move forward.

Getting started was easy and I really liked the admin interface, but about five minutes into it, I realized Django wasn’t going to work for me. There was only one reason: the tightly coupled ORM. Sure, I could just not use that part and use a different ORM, but somehow that seems “impure.” Maybe that seems silly, but remember that I had only decided to use Django on, essentially, a whim.

The reason I don’t want to use the built-in ORM is, I want the Web interface to be orthogonal to the core functionality. I want them completely separate, as they are now in the CGI version. I want to keep database access in the back end and be able to create other interfaces as necessary. Now, I suppose I could just use the Django ORM in the back end (?), but there’s another ORM you may have heard of, SQLAlchemy, that I was/am interested in trying out.

This led me back to Pylons and thinking about some of its good points: it’s WSGI from the start, Routes seems like a really cool way to map URLs, and the (Myghty) templates use Python.

Thinking about it, Pylons seems more like the “natural” choice for this project and its architecture. On the other hand, I think Django would be great for projects that don’t need such separation between the front and back ends (or where there is no back end as such).

I’ve just gotten started with the CGI=>Pylons conversion and so far it’s been a pleasure. I’ll report in more detail once I’ve gotten further into it. One “gotcha” that came up, that I don’t think this is specific to Pylons, was creating a controller with the same name as the top-level package. Don’t do something like this:

paster create –template=pylons tripplanner
paster controller tripplanner

Python will be unable to import the controller.

[1] The creator of Python (which I mention for non-technical readers)
[2] vpslink.com (seems good so far)