CREATE TABLE b (p PERIOD);
ALTER TABLE b
   ADD EXCLUDE USING gist (p WITH &&);
INSERT INTO b
   VALUES('[2009-01-05, 2009-01-10)');
INSERT INTO b
   VALUES('[2009-01-07, 2009-01-12)'); -- ERROR

Postgres 9 added Exclusion Constraints opening up more ways to specify UNIQUE constraints on a table. Adding in the temporal “period” datatype for Postgres, you can create a constraint that prevents any two periods from overlapping. There’s simply no way to enforce that with a constraint without that support so most of the time we enforce it using application logic.

Let’s take for example an application that schedules professors to classrooms. No two professors can teach in the same classroom at the same time: there must not be any overlap. Here’s a couple lines of SQL that would prevent invalid data:

CREATE TABLE reservation(room TEXT
        , professor TEXT
        , during PERIOD);

-- enforce the constraint that the
-- room is not double-booked
ALTER TABLE reservation
    ADD EXCLUDE USING gist
    (room WITH =, during WITH &&);

-- enforce the constraint that the
-- professor is not double-booked
ALTER TABLE reservation
    ADD EXCLUDE USING gist
   (professor WITH =, during WITH &&);

The reservation table has two exclusion constraints on it which prevent overlap for any single room and any single professor. The net result is that room-professor bookings are guaranteed unique no matter how long a given classroom is booked (some classes might be 60 minutes and others might go all day).

Combined with the Developing Time-Oriented Database Applications in SQL, this makes for some interesting brainstorming. Also read the Exclusion Constraints post.

Once home I searched for a database of conjugated verbs to make flash cards that, rather than working with infinitives, would read simple actions like “They walk”, “He used to sing” or “We would have spoken” and the reverse would have the proper Spanish conjugation. Despite my uber Google skills, I was unable to find any non-commercial products. However, I did come across one great resource that had the data I needed.

Fred Jehle, formerly a professor at Indiana University-Purdue University Fort Wayne, published approximately 600 verbs, fully conjugated in all moods and tenses, on his website in 1998. The resource helped students improve their verb use in addition to a variety of notes on other aspects of the language. I contacted Mr. Jehle to inquire if a database of his verbs were behind the scenes but unfortunately only the static web pages exist.

Out of curiosity, I opened up a couple of pages to see what the source HTML looked like and, luckily, it was pretty uniform. I broke out my editor and wrote a script to read in each page, parse out the various conjugations and dump them in to a (PostgreSQL 9.x) database. The roughly 600 verbs converted to 11,467 combinations of moods + tenses.

In coordination with copyright holder Professor Jehle, this data is available free of charge via a Creative Commons license for anyone to use for non-commercial purposes so long as you provide attribution. If you alter, transform or build upon this work then you may distribute the resulting work only under the same license.

My thanks go to Mr. Jehle for quickly answering my questions and allowing me to publish the data for other would-be Spanish students. I recommend that you also check out his website for additional Spanish content at http://users.ipfw.edu/jehle/VERBLIST.HTM.

Now, to the script:

#!/bin/bash

# init vars
BACKUP_ROOT=/dblogs/backups
S3CMD=/usr/bin/s3cmd
S3DIR=s3://your-bucket-name/your-folder-name

# take the last X backups and make sure they are on S3 (but not older ones)
BACKUP_COUNT=5
BACKUP_FILELIST=/tmp/.pgbackup_s3cmd_filelist

# first list out the files we want to sync up
ls -1 $BACKUP_ROOT | tail -$BACKUP_COUNT > $BACKUP_FILELIST

# now sync them; requires trailing slash to sync directories
$S3CMD sync --progress --delete-removed --acl-private --exclude '*' --include-from $BACKUP_FILELIST $BACKUP_ROOT $S3DIR/

# clean up
rm -f $BACKUP_FILELIST

Setup s3cmd after installing it by running s3cmd --configure with your Amazon credentials handy. I also use s3fox, a Firefox add-on, as another way of quickly accessing S3 with a GUI.

This script maintains only the last 5 backups on S3 and it deletes what would be the 6th backup each night via cron. You could adjust the number to keep with the BACKUP_COUNT parameter. I’m personally using this to backup a PostgreSQL server but you can adjust it to back up any type of directory with files in it.

Update 2/17 – realized the default behavior is not to delete files on the remote host that are not included locally. I updated the above to include the –delete-removed argument and now it deletes the oldest file as expected.

Big changes include truly integrated hot standby and streaming replication (without using third party tools) and more than 200 total improvements. We’ve been kind of lagging behind with 8.3 but it looks like it’s time to schedule an upgrade window…

For Linux, HP has a tool available called hpacucli (HP Array Configuration Utility for Linux) for interrogating HP/Compaq array controllers (SmartArray 5i, 6i, whatever) from the command line. Before you can install the RPM (on CentOS/Redhat), you will need to first install a compatibility library:

yum install compat-libstdc++-296
rpm -Uvh hpacucli-8.0-14.noarch.rpm 

Then I put this snippet into a new file /etc/cron.hourly/raidstatus:

#!/bin/sh
/opt/compaq/hpacucli/bld/hpacucli ctrl all show config | egrep -i "(fail|error|offline|rebuild|ignoring|degraded|skipping|nok)"

The command /opt/compaq/hpacucli/bld/hpacucli ctrl all show config normally generates something like this (from our development database server):

Smart Array XXXXXXX in Slot 0      ()

   array A (Parallel SCSI, Unused Space: 0 MB)

      logicaldrive 1 (33.9 GB, RAID 1+0, OK)

      physicaldrive 2:0   (port 2:id 0 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:1   (port 2:id 1 , Parallel SCSI, 36.4 GB, OK)

   array B (Parallel SCSI, Unused Space: 0 MB)

      logicaldrive 2 (67.8 GB, RAID 1+0, OK)

      physicaldrive 2:2   (port 2:id 2 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:3   (port 2:id 3 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:4   (port 2:id 4 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:5   (port 2:id 5 , Parallel SCSI, 36.4 GB, OK)

I believe you can reduce the grep to just “(fail|nok)” but I’m taking the conservative approach here. Change the permissions to 0700 and if you have SELinux running make sure the context is set properly.

If your array and controller are in fine shape, then this command will output nothing. If you have a dead drive, it will generate content which will cause cron to mail the root user about it. Bingo – time to go to the colo!

I have seen other people use “ctrl all show status” which generates:

Smart Array XXXXXXX in Slot 0
   Controller Status: OK
   Cache Status: OK
   Battery Status: OK

I prefer to query the config which looks at individual physical drives in addition to the status of the array. I have seen cases (just last week) where one dead drive in the array still lists the array status as OK (because, technically, it is OK, it’s just not optimal and may be pending major disaster!)

Update 5/25/2011 Had an error today, needed this code and corrected a few things. I fixed a typo for a missing quote in the raidstatus script above and added a link to the actual utility. For reference, the output from an error looks like this:

physicaldrive 2:2   (port 2:id 2 , Parallel SCSI, ??? GB, Failed)
physicaldrive 2:5   (port 2:id 5 , Parallel SCSI, 72.8 GB, Rebuilding, active spare)

INSERT INTO foo (column1, column2)
VALUES (val1, val2), (val3, val4), (val5, val6);

This wraps more than one row into a single commit and speeds things up. I’ve seen claims of about 3x performance on Postgres and as much as 10x on MySQL. The good news is wrapping inserts into this format is simple. Here’s some code to get you started for looping over an array of arrays (converted by Ben Nadels’ very quick CSVToArray):

< !--- how many at a time to commit --->
<cfset incr = 1000 />

<cftransaction>
<cftry>

< !--- ii = 1, 1001, 2001, 3001, 4001, ... --->
<cfloop from="1" to="#len#" step="#incr#" index="ii">

  <cfquery name="insert" datasource="#dsn#">
    INSERT INTO someTable (field1
			,field2
			,field3)
    VALUES
	< !--- 1, 2, 3, ... 1000;  1001, 1002, 1003, ... 2000; ... --->
	<cfloop from="#ii#" to="#min(len, ii+incr-1)#" index="jj">

	  < !--- prevent trailing comma error --->
	  <cfif ii NEQ jj>, </cfif>
	  (<cfqueryparam value="#arrData[jj][1]#" cfsqltype="cf_sql_char" />
	  ,<cfqueryparam value="#arrData[jj][2]#" cfsqltype="cf_sql_varchar" />
	  ,<cfqueryparam value="#arrData[jj][3]#" cfsqltype="cf_sql_date" />)
	</cfloop>
  </cfquery>

</cfloop>

<cfcatch type="any">
	<cftransaction action="rollback" />
	<cfrethrow />
</cfcatch>

</cftry>

</cftransaction>

This groups the INSERTS into bunches of 1000. In my code, I wanted the records to either all commit or all fail so I wrapped it in a transaction; you could surely remove that and the corresponding try/catch for your own needs.

Grouping the inserts was taking about 90ms per 100 records over a total of 4,720 test records. The one-insert-per-query approach was taking around 270ms per 100 records. I suspect, run directly against the database as opposed to through ColdFusion and JDBC, there would be less absolute gain but the relative result is about a 3x improvement in this small test case on my development laptop. The database is PostgreSQL 8.3 but this technique also works with MySQL.

Enter RubyRep, a Ruby-based replicator that already supports both Postgres and MySQL and has plans to also support Microsoft SQL Server, Oracle and IBM. The author has published a screencast where he enables and demonstrates working replication in less than 5 minutes.

It handles composite primary keys, single primary keys, sequences/uniqueids and even tables without primary keys with a touch of configuration. It works using triggers which it installs and maintains automatically. There are some limitations with MySQL since it only supports one trigger per table which could collide with an existing application trigger. We’ll have to see what performance is like but it’s pretty compelling so far.

Postgres 8.4 RC1 is out too – might be a good opportunity to roll out a test environment and start playing.

• Look at replacing my validate() methods with Bob Silverberg’s ValidateThis framework which has centralized validation rules. One thing I’ve struggled with in centralized validation is how to handle the situation where a user and an admin have differing validation requirements depending on who is doing the editing. Bob solves this with configurable “contexts” which look promising.
• There are a few places in my system where I am working with collections of ~100-300 objects and using Transfer to manipulate these can be slow. Peter Bell’s Iterating Business Object, which is simply a fancy wrapper for a query, is a performant way of working with collections of this size. The fact that we, in CF land, have to worry about this makes object oriented design a compromise but that’s the way it goes. It’s time to evaluate this as an option.
• Refactor to Abstract Classes. Bob Silverberg gave a presentation on building an object oriented model which included discussion of Abstract Classes. I have used one as the basis for my Transfer decorators for some time which defines a generic populate() routine (influenced by Bob in the past) but it’s time to look at these for the remainder of my objects in an effort to reduce my total code footprint and centralize the similar code. Because I refactored from a query-based framework, my gateways and services leveraged a lot of that code and I wound up with gateways that have their own queries and services that repeatedly proxy calls to those gateways along with calls to Transfer for persisting my objects. One of the things I mentioned in my frameworks migration is starting with my original code may have been a limitation in my migration and the kind of blind cut-and-paste that is time efficient but not particularly elegant is one of those places. Now it’s time to go back and optimize.
• ColdFusion 9 CFC performance is supposed to be substantially better – I have the beta so I need to do some head-to-head tests of Transfer on CF8 vs. CF9 as well as Transfer on CF9 vs. the built in Hibernate support. Gains here may make an IBO unnecessary.
• Get my Trac install updated. I’m still running some ridiculous 0.9.5 version while the latest is up to 0.11.4. Getting 0.9.x Trac to install and run was such a dependency-based nightmare that I swore off touching it but that was 3+ years ago and I’ve got a new server being built to replace the hardware anyways. This older version is not only compromising the feature set but it’s also preventing me from updating my Eclipse setup to take advantage of Mylyn so I can work more from Eclipse against my outstanding tickets.
• I sketched out the definition for a reporting overhaul we’ve been talking about for a year. I already have a pretty sophisticated CF-based reporting system that emulates a lot of what you would get out of a Crystal Report or CFREPORT but does it without requiring the end user to install a report builder tool or have any knowledge beyond the data they’re working with. I’m interested in exploring BIRT further but the user interface for generating reports must be user-friendly. Actuate’s version of BIRT has a very nice UI but costs some dollars.
• The Writing Testable Code: Real-World TDD session by Marc Esher was a great kick in the butt to improve my unit testing. The way he took some very reasonable code and demonstrated how to refactor it to be more testable (and thus, more reliable) was eye opening despite the fact that it was simple. Sometimes the best ninja tricks are. And, he finally explained concisely how to use a mock object in testing that I grokked so it’s time to implement some of these ideas.

Presentation Slide decks

As promised, I’m publishing the slides from my presentations for any attendees who want to grab the details. Thanks for coming and be sure to fill out the CFO survey. The slides, particularly the Postgres preso, were bullet-point heavy which is not my favorite style of presentation (and when compared to say, a Marc Esher presentation, is a bit embarrassing) but when you are walking people through the features of a package, I consider the feature-list-with-commentary to be the equivalent of showing-code-with-commentary. You don’t use pictures to represent your code, now do you? The framework migration slides include my recommendations for surviving a migration of a code base but spare the reader my personal pain and suffering from my launch last year. You only get the actionable stuff and none of the emotional scarring! Don’t forget: build a functional equivalent at all costs!

Move over MySQL, Make Room for PostgreSQL
ColdFusion and MySQL have been best friends since MX and many developers use it as their day-to-day database. Brian made the leap to PostgreSQL in 2004 and never looked back: it has everything MySQL has plus most of the enterprise features found in Oracle, SQL Server and DB2 wrapped in an open source package with a great community and support. Come learn how these packages are more similar than different and why your next project should be backed by Postgres.

Download Slides (PDF)

Planning your migration to a Framework
Still waiting to make the jump to an object-oriented framework like Model-Glue, Mach-II or Coldbox? As the sole developer and survivor of a seven-month migration from a home-grown framework to Model-Glue, Coldspring and Transfer, Brian will share four key areas where unexpected complexity resulted in catastrophe and how you can avoid them in your own applications. This is your survival guide for moving code to a new framework.

Download Slides (PDF)

If you have any questions or feedback about my presentations, feel free to hit me up here.