For a long time now whenever I’m on the search for something interesting to work on that could potentially be of value to myself and others my attention is brought to topics like machine learning, data mining and natural language processing.

These are all areas within computer science which are not particularly easy and have a higher barrier of entry compared to building a typical web application for instance. Yet, if implemented correctly they can provide great insight and fame and fortune are bound to be near.

Joking aside. I studied artificial intelligence at university and my undergraduate dissertation was focussed on expert systems so I’ve a bit of exposure to these topics but nothing decidedly practical nor within the past decade.

Interesting Books

Some searching on Amazon turned up a couple of books which looked to be talking about the type of tasks I was interested in performing.

One thing these books have in common is that they use code examples in Python.

For nearly half a decade now a lot of my interest has been with Ruby and related technologies and so I’ve largely avoided Python considering the similarity of the space the two languages sit in. A look through the most popular AI books reveiled none using examples written in Ruby however.

The other day I came across a copy of another book using Python so I thought I may as well get a bit of hands on experience with the language and take it from there.

Something Practical

In the past I’ve tried to teach myself many languages from books but it has never been until I’ve needed to use the language for something practical that the know-how stuck.

Thankfully I didn’t have to wait long until inspiration struck. As a private investor in the current economic climate, a topic that has been in my mind has been the rate of inflation. Everytime I want to check how quickly my assets are being eroded by inflation I have to unleash Google and hope I land on the correct page and from there find the current rate amongst all the other information. This seemed like the perfect fit for yet another single serving web app.

My starting point was to write a Python script to scrape the Office for National Statistics for the current value for the Retail Price Index.

Next up was creating a basic web-app with App Engine. I’d tinkering with App Engine a bit previously so it was relatively straight-forward to follow the tutorials and piece together a single page app. Slightly more complicated was reading up on the Datastore and Memcache APIs but there were plenty of examples to follow.

Conclusion

The end result is ukinflation.appspot.com. It’s basic but deals with the job at hand. It’s also been a good introduction to various components of the App Engine environment: Datastore, Memcache, Cron, templates and the built-in webapp framework.

I’ve used in all of these types of technologies in other forms over the years and it’s amazing that you get so much functionality out of the box and as long as you embrace the constraints, scaling out comes for free.

I’ve still to make my mind up about Python though, maybe I’m just too accustomed to Ruby. Still, I’m glad to have another tool in the toolbox and to not be restricted to any one set of tech.

Share the Love

As always the code is available on GitHub, fork away!

Tonight I thought I’d have another quick look at how it works, what it takes to build a Rack-app and what information about the request is available to the handler.

The result was the following, an object which responds to call and returns a HTTP response code, headers and a body:

ip = lambda do |env|
  [200, {"Content-Type" => "text/plain"}, [env["REMOTE_ADDR"]]]
end
 
run ip

I have this running on ip.stevenwilkin.com and it provides the same useful function as whatismyip.com but without the ads

The code is available on GitHub as always.

A few months ago I was tasked with migrating and maintaining a bunch of legacy Rails apps and a couple of them were misbehaving. Requests were hanging which was tying-up the front-end Apache child processes and resulting in all the web-apps on the server becoming unresponsive

At the time I didn’t know what was causing the hanging requests, it wasn’t happening in a predictable manner and on the surface I had all the apps dependencies in place.

Seeing as this was a job I was doing on the side I had very limited time to investigate the underlying issue and decided I would have to make do with just knowing when the apps went down so I could ssh in and manually kill off the offending processes.

My initial thought was to use Nagios for monitoring but it seemed excessively heavy-weight for the task at hand so I seized the opportunity to quickly develop a tool perfectly tailored to my needs.

The resulting script, minder, checks a list of domains and if the root of each can’t be read within 10 seconds a notication is sent via XMPP. Nothing more, nothing less. Simple.

Minder proved effective and I was bombarded with instant messages whenever any of the apps went under. Thankfully I found to bit more time to investigate more thoroughly and tracked the issue down to a missing TrueType font used by ImageMagick.

The first iteration of minder was very simple but I thought I’d put a bit of extra effort in to make it easier to customise and share it with the internet-at-large, hopefully someone will get some use out of it.

Prerequisties

• Two XMPP accounts:

  • one to send notifications from, and
  • a second to receive notifications

  Google Talk or any regular Jabber/XMPP accounts will suffice

• Blaine’s xmpp4r-simple gem must be installed:

  sudo gem install xmpp4r-simple
  

Usage

• copy minder.yaml.sample to minder.yaml
• edit minder.yaml to specify the username and password of the sending XMPP account
• specify your own personal XMPP account (xmpp_to)
• modify the list of domains to monitor
• run the script

  $ ./minder.rb
  

• Optionally run the script from a Cron job, the following will run every hour:

  0 * * * * /path/to/minder.rb > /dev/null 2>&1
  

Get the code

Clone the repo from GitHub. Fire any queries to @stevebiscuit.

At that time I’d already started tinkering with apps to track my efforts in the gym and changes in my bodyweight, I’ve used notebooks for this for years and thought it’d be interesting to have this information in electronic form.

Sharpening skills

While I was on the bench waiting for my current contract to start I was sharpening my skills through developing some small projects with the technologies I’d be using. One of those technologies was SQLite which I knew of but didn’t have much hands on experience with.

I started by playing with the Ruby interface to SQLite, using it to insert and retrieve data from a simple database. I’ve since grown to love the simplicity and flexibility of SQLite and will no doubt use it again in the future.

I needed a practical application and thought back to my intentions to track my online activities. Now was as good a time as any to to tick this off the list of potential projects.

When I’d glued together some Ruby to parse the RSS feed of this blog I knew the basics were in place so I started on a simple Sinatra app to act as a web interface.

Stumbling block

I scoured the web for design inspiration and got a good start to the visual aspect of this project but couldn’t make my mind up over a few subtle items. Next thing my agent called: the purchase order finally came through and my contract was to start the next day.

The demands of adjusting to a new work environment meant I felt little motivation to work on anything on my own time and I became more interested in funneling my excess income into the stockmarket. The project joined those other half-finished unfortunates.

A fresh jolt of motivation

After a couple of months on the contract I started feeling a renewed motivation to get something of my own out there. I couldn’t bare the thought of starting another project to not finish it so I set out to to whip this latest app into shape and release it as soon as possible and no later.

I dug through the uncommited changes in my working copy and made some arbitrary decisions which resulted in a design I felt was “good enough.” I modified my RSS parsing script to feed into SQLite and extended it to process my Delicious bookmarks and loved tracks on Last.fm. I created pagination links, wrote a README and sorted out the hosting and cron jobs. Bingo.

Check it out

My original intention was to track much more of my online life and there’s some details I’d like to change but I love releasing code and I’m pleased with this first iteration.

The app is hosted at life.stevenwilkin.com and the code is available on GitHub, I hope you like it.

Deliberate Practice

It’s only a trivial app but I’ve a bit of free time before the start of a contract so I thought it would be good practice to redevelop it, a task I’d been ignoring for a long time.

Goodbye CakePHP, Hello Sinatra

The original app was put together quickly with CakePHP, a PHP-based framework I’ve used since my first paid programming gig. Since then though my interest and proficiency with Ruby and it’s frameworks has come more to the forefront. By choosing Sinatra, the app was given access to the rich Ruby ecosystem, including the joyful HAML and SASS.

The interesting aspect of this release is that all the logic is now implemented on the client-side using Javascript. Originally the countdown was calculated on the server-side and passed to the client with an AJAX call, but the server is more-or-less just hosting the markup, stylesheets and Javascript files required to get the app up and running in the browser.

Heroku

As this app doesn’t make use of a database or Cron jobs etc I thought I’d take the opportunity to move it from my own infrastructure and onto the awesomeness of Heroku whose free offering is a lot more performant than my little VPS. Heroku is a service I’ve been playing with for a while now and it makes deploying a Ruby web app into production a breeze, assuming your project can work within it’s limitations.

All that’s left now is for the DNS changes to propagate. I’d like to show a bit more love to this app in the future, the controls to specify your own home time could be more intuitive for instance, but for now I’m content with the improvements I’ve made to it and it’s new home.

Is it passed your home time yet?

Javascript outside of the browser?

This concept seemed almost ridiculous to me at that time but I thought it was worth a look and cobbled together a quick “Hello World.” I was greatly amused by creating a command-line app with Javascript but did nothing more about it until last week. I’m currently starting out as an independant contractor & have a bit of spare bandwidth so I thought I’d put together something a bit meatier in order to learn more about this system & to also sharpen up my Javascript skills.

Dave had shown me node-comment and I also studied node_chat. From there I had just enough reference material to understand the mechanics of long-polling and to start piecing together a simple real-time commenting system.

What is node?

In and of itself, node is not a web development stack. It provides an I/O platform of which HTTP handling is a part. People are building blogging and wiki engines, FTP servers and command-line tools with this thing, anything I/O based is possible and thankfully micro-frameworks like (fab) can ease the pain of creating web apps.

Node is still very young software, which I have felt the pain of. To serve up static content I wanted to use paperboy as it can be used as middleware within (fab), however mere days after the last update to the project the API it was relying upon in node changed fundamentally. Despite being new to node development I was able to patch paperboy to work with node v0.1.31 and get on with my actual project. Another sticking point was the inability of (fab) to handle POSTed form variables, this functionality is planned for it’s next release though.

Fun

Ultimately, my app is of little practical use: it doesn’t make use of persistance or moderation and node’s suitability for the production environment may itself be questionable at this time, but it was quite a fun experience. From something I more-or-less despised a decade ago I’ve come to develop a real affection for coding Javascript over the past few years.

When I had the basics of the app laid-down I eagerly awaiting the moment when I could have it open in two browser windows and enter a comment into one and see it appear in the second. That moment came and it brought a smile to my face. I suggest you give it a go

Get the code

As a wise man once said, code speaks louder than words, so please be my guest and clone my demo on GitHub.