Archive for the 'Technical' Category

Scroll Wheel Zooming Added

Published on April 20, 2007 in Google Maps, Python, Technical, Trip Planner and Web Development. 0 Comments

A few days ago Google added scroll wheel zooming to their public maps API. Today, I enabled that feature in the trip planner. It was really simple, only requiring the addition of one line of JavaScript code.

This feature has been available on the official Google Maps site for a while, but it wasn’t available to third party developers until three days ago [1].

Another feature Google released recently is the ability to encode long lines for more efficient rendering. For long routes, this means the annoying “this script is taking a long time, do you want to continue popup” shouldn’t come up again.

We haven’t incorporated this into the trip planner yet but will some time in the next week or so (it’s a bit more complicated than adding scroll wheel zooming). I wrote some code that does the line encoding; now I just need to set up the back end to encode lines before sending them to the Web.

[1] Google Maps blog post about scroll wheel zooming

Upgraded to Python 2.5

Published on February 11, 2007 in Core, Python, System Administration, Technical and Trip Planner. 0 Comments

I just got done installing Python 2.5 and converting the trip planner code to use it instead of 2.4. The only required change was pointing the “dispatch” file at the 2.5 executable. No other changes to the code were required to make it run on 2.5.

A few little things also got cleaned up, and I updated to the latest versions of MySQLdb (1.2.2b3) and simplejson (1.5).

Hopefully nothing will blow up.

Using PostGIS with SQLAlchemy

Published on January 29, 2007 in Core, Data, GIS, PostGIS, PostgreSQL, Pylons, Python, SQLAlchemy, Technical and Trip Planner. 1 Comment

The development version of the Trip Planner uses a Postgres/PostGIS backend (instead of MySQL) SQLAlchemy as the ORM (instead of raw SQL), and PCL for Python geometry types (instead of our own ugly hacked versions).

Question: How do you move geometries out of Postgres/PostGIS into PCL types via SQLAlchemy and vice versa?

Answer: Create a custom geometry column type.

Here’s our SQLAlchemy geometry type definition and subtypes (points, linestrings, and multilinestrings; adding other types should be trivial):

sqltypes.py

Look here for an example of using it:

tables.py

The type is pretty simple. The only tricky part was figuring out how the database stores the geometry. It’s in ASCII hex, so we use binascii.a2b_hex to get a binary representation and feed that to the PCL fromWKB factory. In the other direction we use binascii.b2a_hex on the WKB representation of the PCL geometry.

[8/7/09: Updated links to source code. Note: latest version uses Shapely, not PCL.]

WordPress CMS

Published on January 25, 2007 in Meta, Site, System Administration, Technical and Web Development. 0 Comments

Previously, we were using a hacked together “content management system” (CMS) to maintain this site. We had integrated WordPress for blogging, but all the pages were stored in separate files. Keeping things updated was a pain in the ass.

In order to simplify things, we wanted to move the whole site and blog into a unified CMS. We tried Mephisto, and it’s pretty nice, but we didn’t want to be tied to using Ruby on Rails, which Mephisto is built on.

We can run WordPress almost anywhere, easily, and inexpensively. There’s also a ton of info for hacking WordPress on the Web. PHP is pretty weak compared to Python or Ruby, but that doesn’t really matter for this purpose.

The theme for the site is being developed by Wyatt and is available for download via Subversion (you might need to log in with username:password guest:guest):

svn co http://svn.bycycle.org/bycycle.org/wordpress/themes/byCycle/trunk byCycle

We may have introduced some broken links or other problems during this change. If you notice anything, please let us know.

Python Cartographic Library (PCL) — Installing Just the Spatial Package

Published on September 25, 2006 in GIS, PostgreSQL, Python, SQLAlchemy, Technical and Trip Planner. 2 Comments

So let’s say you need a spatial geometry library for Python. You could write your own; you could also use the PCL. The PCL includes some packages we don’t need, like one for MapServer rendering. I only installed the minimum needed to get the spatial package working (which I’ll talk about below).

[Note: "geometry" refers to points, lines, polygons and other geometric forms used to represent real-world objects. Examples: intersection (point), street (line), zip code boundary (polygon).]

I wrote a rudimentary geometry library for the trip planner that’s been working fine, but now I need to do some more “advanced” stuff related to using PostGIS and SQLAlchemy. In particular, I want to convert database values to Python objects and vice versa.

The first part (database to Python) is fairly easy and our current library already does that, but it’s convoluted in that it gets the database value as well-known text (WKT), parses that, and creates a Python object. From what I can tell, the PCL can go straight from well-known binary (WKB) geometry to Python objects.

The second part (Python to database) is harder because it involves converting a Python object to a binary geometry value. I don’t know anything about the binary geometry format and I don’t want to know, and it looks like with the PCL I don’t need to know.

I’m assuming PostGIS and PCL will get along together because they both rely on the same libraries, proj4 and geos. We’ll see.

The installation was fairly straightforward. The PCL includes five sub-packages. We had to install two of them, PCL-Referencing and PCL-Spatial. PCL-Referencing requires proj4. PCL-Spatial requires PCL-Referencing and geos >= 2.2.2. Something in there also requires the OGR library, which is included with GDAL.

The basic steps are, install proj, geos, and gdal, then install PCL-Referencing, and lastly install PCL-Spatial. On Ubuntu 6.06 (Dapper), here’s what I actually did:

  • Installed proj4 using apt-get
  • Installed libgdal using apt-get
  • Installed geos 2.2.3 from source into /usr/lib. I installed this over a slightly older version of geos installed using apt-get; hopefully that won’t cause any issues.
  • Checked out the PCL trunk:
    svn co http://svn.gispython.org/gispy/PCL/trunk PCL
  • Installed PCL-Referencing with the usual python setup.py install
  • Installed PCL-Spatial with the usual…

Test Driven Development, PostgreSQL, SQLAlchemy

Published on September 13, 2006 in Core, Data, PostgreSQL, Python, SQLAlchemy and Technical. 1 Comment

I just tried out some o’ that new-fangle “Test Driven Development” (TDD) I’ve been hearing about. Yeah, it’s good stuff.

At the moment, I’m in the process of migrating GIS data from MySQL to PostgreSQL so we can take advantage of the PostGIS spatial extensions. I’ve also been making a bunch of related changes (AKA refactoring) in the “model”, separating things that never belonged together, and so forth.

I started out by rewriting the MySQL data import script. [Note: basically, the script pulls data out of a flat ESRI shapefile and normalizes it.] This wasn’t a complete, from-scratch rewrite–a lot of stuff I just copied over and tweaked a little bit. The biggest changes here were due to using SQLAlchemy instead of typing out SQL queries. I’ll just note that SQLAlchemy is “da bomb” and makes many things easier (once you get the hang of it).

That part was pretty straightforward, and low-level–I didn’t get into the ORM aspects of SQLAlchemy at all.

The next step was to modify the routine that creates adjacency matrices for routing. In the end, this was straightforward too. I ended up reusing some stuff from the new import script, which was cool. I refactored a lot during this process, adding some new modules and classes.

[Here's where we get to the TDD aspect.]

So, I was sitting there (here really) thinking, “Hmmm… what now?” I drew some diagrams with the new classes and associations…. OK, that’s fun…. “Wait, I know. Run the unit tests!” Doy!

I started with the address normalization service, since the other services both depend on it. The test suite for this service isn’t as comprehensive as it probably should be (there are 19 tests), but it proved to very useful for shaking out a bunch of bugs in all that refactoring. The tests also helped keep me focused, and that aspect might be more important than the bug-squashing aspect (maybe).

Today, address normalization. Tomorrow, geocoding.

Foray Into Pylons (Python Web Framework)

Published on August 17, 2006 in Pylons, Python, Technical, Trip Planner and Web Development. 1 Comment

For a while now I’ve been planning to start using one of the many available Python Web frameworks in the development of the trip planner. Over the course of the project we have built an ad hoc, application-specific, CGI framework (of sorts). Using a “real” framework will allow us to stop refining those wheels we reinvented and instead focus more on the specifics of our application.

Deciding which framework to use has been tough. In the past year or so it seems Web frameworks have become all the rage, with several interesting Python projects and all the Ruby on Rails hype. I had initially narrowed it down to two choices, having researched just about all the competitors: TurboGears and Django. I don’t remember Pylons in that first round of research, but maybe I just missed it. Once I noticed it, it was added to the list too.

So, the idea had been floating around for a while and I had created several test applications with various frameworks but still hadn’t gotten around to making a definite decision and switching over. Recently, I listened to a podcast interview of Guido van Rossum[1]. In it he mentioned that Django is his favorite Web framework. He didn’t say much about why except that he likes the way the project is being run. (“They really get open source.”)

Having heard that and having also narrowed the field to just Pylons versus Django, I decided, “OK, well I’m going to go ahead with Django then.” I ordered up a new Ubuntu VPS[2] to host the new version of the trip planner on and got to work. It was exciting to finally make a decision and move forward.

Getting started was easy and I really liked the admin interface, but about five minutes into it, I realized Django wasn’t going to work for me. There was only one reason: the tightly coupled ORM. Sure, I could just not use that part and use a different ORM, but somehow that seems “impure.” Maybe that seems silly, but remember that I had only decided to use Django on, essentially, a whim.

The reason I don’t want to use the built-in ORM is, I want the Web interface to be orthogonal to the core functionality. I want them completely separate, as they are now in the CGI version. I want to keep database access in the back end and be able to create other interfaces as necessary. Now, I suppose I could just use the Django ORM in the back end (?), but there’s another ORM you may have heard of, SQLAlchemy, that I was/am interested in trying out.

This led me back to Pylons and thinking about some of its good points: it’s WSGI from the start, Routes seems like a really cool way to map URLs, and the (Myghty) templates use Python.

Thinking about it, Pylons seems more like the “natural” choice for this project and its architecture. On the other hand, I think Django would be great for projects that don’t need such separation between the front and back ends (or where there is no back end as such).

I’ve just gotten started with the CGI=>Pylons conversion and so far it’s been a pleasure. I’ll report in more detail once I’ve gotten further into it. One “gotcha” that came up, that I don’t think this is specific to Pylons, was creating a controller with the same name as the top-level package. Don’t do something like this:

paster create –template=pylons tripplanner
paster controller tripplanner

Python will be unable to import the controller.

[1] The creator of Python (which I mention for non-technical readers)
[2] vpslink.com (seems good so far)

WordPress Upgrade and K2

Published on August 13, 2006 in Meta, Site, System Administration and Technical. 0 Comments

Today we upgraded to WordPress 2.0.4. It’s a security release. Upgrading is easy: just copy the new directory over the old (being careful to save any customizations first, although if you created new directories for your custom themes, you won’t have to do anything).

We also moved from the old default theme, Kubrick, to a new version of it called K2. This version makes a lot of things easier, and it looks better out of the box. One day we might customize it.

Installing Trac

Published on August 5, 2006 in Site, System Administration, Technical and Trip Planner. 1 Comment

Today I installed Trac. “Trac is an enhanced wiki and issue tracking system for software development projects.” We plan to use it to keep track of issues with the trip planner and any other software projects we might undertake.

Issues (AKA bugs) with the trip planner can now be reported at http://trac.bycycle.org/newticket. This is somewhat developer-oriented, but users can (and should!) use it too.

Also, since Trac contains a Wiki, anyone can contribute useful information about the trip planner, which might be documentation, tips and tricks, or anything else that’s relevant.

Here is the high-level outline of our Trac installation:

  • Create a directory to contain Trac projects; each project will live in a subdirectory
  • Create a subdomain at trac.bycycle.org for accessing Trac projects
    • The files for this will live in the html subdirectory of the main Trac directory
    • Each project will have a subdirectory in the html directory that contains a simple script to make the project accessible from the Web

Dependencies

  • Subversion 1.3.2
    • Berkeley DB 4.4.20
  • ClearSilver 0.10.3
  • SQLite 3.3.6
  • PySQLite 2.3.2 (Python bindings for SQLite)

Install Subversion

  • First install BDB
    • cd build_unix
    • ../dist/configure –prefix=$HOME
    • make && make install
  • ./configure –prefix=$HOME –with-berkeley-db=$HOME
  • make && make install
  • cd subversion/bindings/swig/
  • make swig-py
  • make install-swig-py
  • echo ~/lib/svn-python > ~/lib/python2.4/site-packages/subversion.pth

Install ClearSilver

  • ./configure –prefix=$HOME –with-python=$HOME/bin/python –disable-ruby –disable-java –disable-perl –disable-csharp
  • make && make install

Install SQLite

  • ./configure –prefix=$HOME
  • make && make install

Install PySQLite

For some reason, this was the hardest thing to install.

  • Edit setup.cfg, changing the values of these variables as shown
    • include_dirs=/home/u6/bycycle/include
    • library_dirs=/home/u6/bycycle/lib
  • python setup.py build
  • py setup.py install

Install Trac

  • python setup.py install

Set Up and Configure Trac

  • Create a trac subdomain at trac.bycycle.org
    • Create ~/bycycle.org/trac — projects will live here
    • Create ~/bycycle.org/trac/html — each project will have a Web-accessible directory here
  • Create a .htaccess file in ~/bycycle.org/trac/html that points all requests to any subdirectoy of trac.bycycle.org to the index.cgi bootstrap program in the subdirectory

Create a Trac Project Environment for the Trip Planner

  • cd ~/bycycle.org/trac
  • trac-admin ./tripplanner initenv
  • Edit tripplanner/conf/trac.ini
  • cd bycycle.org/trac/html/tripplanner
  • cp ~/share/trac/cgi-bin/trac.cgi .
  • Edit trac.cgi, adding these two lines at the very beginning of the try clause
    • import os
    • os.environ['TRAC_ENV'] = ‘/home/u6/bycycle/bycycle.org/trac/tripplanner’
  • Create a simple c program, index.c, that puts LD_LIBRARY_PATH=/home/u6/bycycle/lib in the environment and then calls trac.cgi; compile it to index.cgi (Questions: why can’t we use Python instead? Why can’t we add that to os.environ in trac.cgi and get rid of index.cgi)
  • Visit http://trac.bycycle.org/tripplanner/ and make sure it works

TODO

  • Serve static files statically instead of via CGI
  • Look into authentication if necessary
  • Add content!

WordPress Setup

Published on July 28, 2006 in Site, System Administration and Technical. 1 Comment

Installing WordPress is straightforward. Since we wanted an embedded blog for news on our front page, along with a standard blog in a subdirectory, we had to go through a few extra steps.

Here’s how to reproduce our setup:

Note: The following assumes Apache, PHP, and MySQL are already installed and configured (correctly), which means, amongst other things, Apache is configured to load the PHP module, there’s a database for the WordPress tables to be installed into, and there’s a MySQL user that can access those tables.

  • Download WordPress 2.0.3 from http://wordpress.org/download/
  • Decompress the downloaded file into the top-level HTML directory; this will create a new directory named wordpress
  • Rename the wordpress directory to blog (not strictly necessary, but looks better)
  • In the blog directory, copy wp-config-sample.php to wp-config.php
  • Open wp-config.php in a text editor and follow the directions in the file (this just involves a few settings for MySQL)
  • Navigate to http://byCycle.org/blog/wp-admin/install.php and follow the directions
  • Assuming Apache, PHP, and MySQL are installed and configured and wp-config.php is correct, the blog is now installed and ready to use at http://byCycle.org/blog/ (Styling the blog to match the rest of the site is left as an exercise)
  • To embed WordPress in the home page, first rename the home page to index.php (index.html => index.php)
  • Add the following PHP to the top of index.php (before anything else):
    <?php
    $cat = 2;
    $show_how_many_posts = 2;
    require('./blog/wp-blog-header.php');
    ?>
  • Determine where the news section will be, probably in something like <div id=”news”><!– News will go here –></div>
  • Inside the news div, copy The WordPress Loop; The Loop can be copied from blog/wp-content/themes/default/index.php; it consists of all the stuff inside
    <div class=”narrowcolumn” id=”content”></div> (but not the div tags)
  • Modify The Loop; this isn’t strictly necessary, but to use the $cat and $show_how_many_posts variables from above, a few changes need to be made; (instead of pasting our version of the loop here, here is the source; look for the section that starts with <!– Begin Embedded WordPress –>)
  • And I think that’s all; let us know if it looks like we left anything out

I should mention that even though PHP is not my favorite language, I think WordPress is pretty cool, and hacking on it is fairly easy.