my name is alan - drupal

Drupal postcode proximity search

alan — Tue, 10 Aug 2010 16:50:50 +0000

my blog_Drupal postcode proximity search

10.08.2010

The brief: a website built around a UK postcode search. The user inputs their postcode, chooses a category to narrow the search if they want, and the results can be displayed as a list of teasers or a GMap. The teaser list needs to be sorted either alphabetically or by distance from the entered postcode. Sounds simple, right?

WRONG

I have always had the impression that the Views module wavers constantly between being incredibly useful and collapsing under the weight of its own complexity. I have used Views to create websites very rapidly, but likewise I have often gotten bogged down in its limitations, bugs and unexpectedly mysterious workings. Often it can be a case of getting 95% of the way there in an hour, then spending 2 days achieving the final 5%, and this was one of those times.

With Drupal 6, Views, Locations and GMap all being quite mature at this point I was taken aback by the problems I encountered accomplishing what I considered to be one of the more basic possible uses of a location search. Searching on Google informed me that I was not alone.

Initially I was going to write about the problems of doing a proximity search from a naked postcode, but the extremely active Drupal community have resolved this issue in the last couple of weeks. So instead I thought I would just write down a couple of the essential elements of the "recipe" that I have used to accomplish my postcode search site, in the hope that someone else out there might find it useful. Who knows, the next release of location or views might make all these comments completely obsolete!

Most of the required functionality came right out of the box with Location Views: once I had a content type associated with a location, and built a few sample nodes, it was relatively easy to create a view listing those nodes. Using "node view" allowed me to style them using Content Templates. I added a location: distance/proximity argument to the view, and this took as an argument the user-entered postcode...NOT! I couldn't work out for a long time why this argument failed to work, until I realized I was missing some small grey "helper text" underneath the settings:

So, OK, I could grumble (and I did) about how this vital info was easy to miss; on the other hand, where were they supposed to put it? Oh yes, that's right - in the comprehensive, helpful documentation for Views...oh. Anyway, let's put this down to developer blindness and move on, now that we know that the postcode argument needs to be country_postcode_distance or postcode_distance (where "distance" is the number of miles away from the postcode to return location results for).

My other argument was a category taken from a hierarchical vocabulary. I tried messing around with exposing a hierarchical-select filter in views, and very quickly ran into some horrible nightmares. To sum up, if I had done it that way there is no way I would have been able to style my form as specified by the graphic designer. So I learned how to add a hierarchical select field to my own form and passed the value of this field into the view as a simple TermID argument.

Now, the user needed to be able to switch easily between Displays in the same view (i.e. viewing the same results as a GMap or a node list) and also switch between alphabetical and distance ordering. Again, after some tinkering with trying to expose filters I realized that my form was going to get very ugly very quickly, so instead what I did was create my own links by passing arguments in the URL. So to display the A-Z list, I added a display to the View whose only difference was that there was an alphabetical sort in the sort criteria, then created a link (nicely styled as specified!) in the form http://mysite.com/myform?postcode=xxxxxx&tid=1&display=page_2. Then using $_GET I was able to choose which display of the view to display, and used views_embed_view($postcode, $tid, $display).

This also enabled the user's choices to be retained while switching between different displays of the view, since I took the $_GET values and also used them to set the values of the form fields. And in one key respect it saved my sanity, namely: sorting by distance.

This should have been easy. I had a view taking a postcode as its origin and calculating distance between the origin and each node listed. However, sorting by the distance proved a severe headache. Options were present to sort by distance from user location, distance from a static postcode, distance from a location-associated nid argument....the full list is here:

What wasn't there was an option to sort by distance from my postcode argument. This one cost me about 2 hours. I quickly settled on the need to use the "Use PHP code to determine latitude/longitude) option, but using standard views syntax to get the views arguments directly ($args[0], $args[1], etc) didn't work. Eventually I figured out that since in my particular case I was passing arguments in the URL, I could use $_GET here like anywhere else to get my postcode. Then I used location_get_postalcode_data() from the Location API reference to get my lat/lon pair. Lastly I just had to figure out that although that array came back in the form array('lat'=>xx.xx, 'lon'=>xx.xx), the return value for the sorting had to be in the form array('latitude'=>xx.xx, 'longitude'=>xx.xx). Live preview didn't work for this one (because of the use of $_GET) but when I saved the view and tried my form again, it worked! Here's the code:

Incidentally, using the Location API was invaluable when I needed to access the distance/proximity value in my Content Template. Again, I used the postcode value from $_GET and passed it into location_get_postalcode_data() to get the lat/lon array. Then I passed both lat/lon arrays (the origin, and the one from $node->locations) into location_distance_between() and hey presto!

These little tricks enabled me to solve my problem and provide a working postal code search facility to my client, but I'm under no illusions about them being a solution flexible enough for anyone else, which is why I'm not providing comprehensive code, just guidelines to other stuck developers about how they might solve their problem. Hopefully a couple more iterations of Views or Location may solve these problems. I had to find a quick and clean-as-possible solution in order to meet a deadline. I found myself caught in the nether world where Views was able to provide so much of the required functionality that it became worth it to try to hack around in search of that remaining 5% rather than coding something from scratch.

None of the above should be interpreted as criticism of the developers who work on the Views and Location modules. These are incredibly complex modules which have to integrate and play nice with a number of other incredibly complex modules, and Drupal itself, so it's no wonder that there are bugs and omissions. It was actually quite exciting to find out that in the 2 weeks since I started development on my project, one of my critical bugs was fixed in a development release!

That concludes my unnecessarily long summary of a developer's solution to a client request. Hopefully someone might find it useful, and if you do, a comment would make me feel all warm and fuzzy!

EDIT #1:I realized I could make my life even easier by calculating latitude and longitude for the origin once (in the form submit handler) and then passing the latitude and longitude as values in the URL. This meant that instead of calling location_get_postalcode_data() 12 times (once in the form submit, once in the Views sort criteria, and once for each node teaser displayed in the View), it only gets called once. Much better.

EDIT #2: Another problem became apparent in testing: the UK postcode data that ships with the Location Module is a reduced set, containing data only up to the area (in the form 'CO1' or 'HG3') rather than the full set of postcode data ('CO1 xxx' or 'HG3 xxx'). When I searched for the full set of UK postcode data, it became apparent why this was: there were over 2 million rows of data to import.

For those interested, I found a Drupal-friendly SQL script for the full UK postcode data here (scroll down to comment #17). However, I found this to be a problem for me. Even once I got it imported, which took quite a while due to MySQL timeouts, I found it slowed my Search page down by a factor of 10 (from average 0.5 seconds to average 5 seconds), and almost all of that time was taken up by the MySQL query used in location_get_postalcode_data(). There didn't seem to be a lot I could do in terms of optimization so I decided to revert to the reduced set of data (which provides a fairly accurate location, it just doesn't go down to the street level and therefore some distance calculations are probably slightly off).

If anyone has a Drupal site, using Location, working with the full UK postcode data set, that isn't slow as a dog when searching that table, then I want to know!

EDIT #3: The client raised a problem that was interesting. After searching by postcode and displaying the View as a GMap, the map was always centred by default on the same location, which was set by a GMap macro in the View settings. The client wanted the map to be centred on the postcode that had just been searched for (a totally reasonable request). However, this wasn't an option. Views offered me the ability to center on a node argument; in other words, if I passed in a nodeid, I could center the map on that node's location. This wasn't very useful to me, since I want to center on the postcode argument, not a node.

One possible workaround occurred to me: on every postcode search, I can create a node, set the node's location to the searched postcode, and then pass that node id as an argument to the View. These dummy nodes could then be deleted on the next cron run. However, this kind of hack should be regarded as a last resort.

I ended up settling for a much more elegant hack which I discovered here. It involves adding 1 line of code to the GMap module which enables the GMap macro field in my view to parse PHP. Once this is done, then it's easy for me to insert some php that reads my postcode (or more accurately, my lat/lon arguments) and outputs the macro using those to center the map.

Of course, now I have to remember to re-insert this line in gmap.module every time I update the GMap module, which is why hacking modules is usually a bad idea. However in this case it was worth it to me, as a) a fix was needed quickly, b) this is obvious and necessary functionality, and c) I am not being retained as the site's maintainer ;-)

It's probably clear from all of the above that Drupal + Views + GMap + Location is almost, but not quite, ready for this extremely common real-world application (searching by postcode).

click image to enlarge_

other recent blogs

Drupal update nightmares

alan — Tue, 15 Jun 2010 03:13:38 +0000

my blog_Drupal update nightmares

15.06.2010

Sometimes things go as planned, and sometimes they don't. That sentence sums up my relationship with Drupal. Tonight was one of the bad times. I just finished bringing my own site up to date with the latest Drupal security release (6.17 at the time of writing), and while I was at it I decided to update all my add-on modules to their latest versions too, because why not? While I was at THAT, I realized that I hadn't put my themes and modules in the right place when I built my site - they were sitting in the core "/themes" and "/modules" directories, which is baaaaaad) - so why not move them to the right place too?

So I did it all at once. But first, just to make sure the subsequent nightmare would be as annoying as possible, I overwrote my local versions of the modules with the new versions, then uploaded all the new ones to the right place on the webserver. Then I deleted the existing modules on the server.

No, since you ask, I didn't take a backup first. Why would I do that? It's only MY website. Some kind of brainfart caused me to skip every single one of the standard precautions I would take 100% of the time if I was altering a client's website. I didn't back up the database. I didn't back up the files. I didn't keep a local copy of the previous files. I just nuked everything and uploaded the new stuff, because at 3am sometimes I am an idiot.

Predictably, a lot of things went wrong. Allow me to list them.

Conflicts with new module versions caused my site to crap out completely with PHP errors.
- Fix: deleted a module I wasn't using anyway
The modules and themes couldn't be found after I moved them from /modules to /sites/all/modules and from /themes to /sites/all/themes. Also, because the themes couldn't be found, I couldn't navigate to admin/settings/performance to clear the cache to fix this.
- Fix: navigated to admin/build/modules which although themeless was still a functional form. Saving the modules page cleared the cache and the modules were picked up again.
Lots of images in my theme didn't show up because I'd hard-coded the URLs in the stylesheet and the page templates and then moved the theme.
- Fix: updated CSS files and content templates with the new image paths.
Javascript gallery stopped working.
- Fix: updated path to jQuery gallery in page.tpl.php
My hacks were lost. This was a real pain. I'd forgotten that as a shortcut, I'd hacked the Flickr and Twitter modules to add some extra tags for style purposes. When I updated to the new module versions, I lost all my hacks. This really was no more than I deserved, since hacking modules is NOT THE DRUPAL WAY. All right, sometimes it's the quickest and easiest way to accomplish something in a hurry, and sometimes in the real world we are in a hurry, but sometimes we pay for our hurry, and in this case it cost me another hour.
- Fix: wrote better CSS this time.

All this trouble taught me a couple of lessons which I thought I knew already, but apparently I needed to learn them again.

Put your add-on modules and themes in sites/all/modules and sites/all/themes
Don't hack core. Don't hack other peoples' modules - or if you absolutely must, don't forget that you've hacked them.
Always take a backup before updating anything.
Don't try to do anything important on a live server after 2am.

Like I said, I stick carefully to the guidelines when working on client sites but took shortcuts for my own, with a kind of mindset that it didn't matter because the only person who'd ever know was me anyway. So this is my confession to the world: I did not follow the Drupal Way, and for my penance I lost three hours of my life. I repent forevermore.

drupal

nightmare

click image to enlarge_

other recent blogs

Favicons

alan — Thu, 17 Sep 2009 05:33:44 +0000

my blog_Favicons

17.09.2009

A favicon is that little icon that appears in the left-hand side of the address bar of your browser when you visit a website, or in your bookmarks menu if you bookmark it, or on the tab if you are using a browser with tabs. Creating a good favicon is an important part of developing a website, since that icon is the most frequently-visible representation of your website. If you design it right, just seeing it will make your site pop into someone's mind when they open their bookmarks. If you get it wrong, or don't bother with it at all, you've missed an opportunity to make your site stand out just a little bit more.

The favicon is often a tiny version of a company's logo, but you can have a bit more fun with it than that. In fact, it's amazing just what people have been able to do within that 16x16 pixel space. You might not consider yourself much of a graphic artist, but the proliferation of online tools for web developers means that you don't have to be. A site I've been using a lot is Favicon.cc, an online favicon generator for static or animated icons. You can import from an image file as a starting point, or design a favicon from scratch. If you have a logo image, then generating a favicon is as simple as importing the logo and clicking a button. Take your new favicon.ico file, drop it into the root of your website directory, and hey presto! The boring default "blank" icon that people saw in their address bars when visiting your site is replaced by something a bit more eyecatching.

As of the time of writing, I've been experimenting with creating animated favicons, and I'm using one on this site. It's a close-up of my eye (imported from a digital photo), with a differently-colored iris in each of the 4 frames of the animation. Slightly cheesy but it made me smile. I've got 15 tabs open in my browser and mine is the only one with a moving favicon, so it stands out.

The main catch is that browsers differ significantly in their implementation of favicons. Internet Explorer doesn't seem to support animation at all, and has problems with transparency, while other browsers, e.g. Opera and Safari, also don't display the animated favicon on my site, and therefore may require different code. If you want to explore this in more detail, you could try clicking here, or here, or maybe here. The good news? The animations work in Firefox, and it's only a matter of time before the other browsers follow suit.

One last note: I work mainly with Drupal, and it's shocking how many Drupal websites out there still use the out-of-the-box default Drupal favicon. It just doesn't look professional in my opinion.

click image to enlarge_

other recent blogs