It turns out there is a bug from JDK 1.7 u75 onwards that is present at least through JDK 1.8 u60 which prevents Jstat, when run as root, from introspecting JREs running as other users.

To solve, simply run jstat as the user who owns the JRE process. You can pass the username to sudo when running jstat like:

sudo -u nobody jstat -gcutil -t <pid> 1s 30

And you’ll be treated to your output again. In my case, I have two server instances running so I like to see their output side by side at the console. Here’s my script for doing that using /usr/bin/paste and some console redirection trickery:

The username can be figured out from the process but this was a quick fix for the above bug.

Thankfully @kdecherf hacked up a wget command to force-feed the proper cookies.

If you want to grab 1.6.0_31, replace the URL in his example with:

http://download.oracle.com/otn-pub/java/jdk/6u31-b04/jdk-6u31-linux-x64-rpm.bin

Note: I’ve already agreed to the terms from my desktop; I just can’t download it to the server.

UPDATE 2016 – @kdecherf’s URL has been updated above for JDK7. I find myself returning to it regularly so I’m capturing a JDK8 download URL here for posterity:

wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie;" http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.rpm

Upgrade to CHKUSER 2.0.9

CHKUSER 2.0.9 includes one update that is very important to us: CHKUSER_DISABLE_VARIABLE. This enhancements allows us to disable CHKUSER checks for certain relays. In our case, our web application servers relay mail and sometimes the TO address is invalid. Without this change, CHKUSER would deny even sending the email so bad emails would queue up on the web servers. What we want to happen is for the mail server to accept them, bounce them and let our bounce handling routines run notifying the sender of the bad address. Here’s the necessary steps:

1. Download development package qmail-toaster-1.03-1.3.21.src.rpm which includes the CHKUSER 2.0.9 code and a few other enhancements (see this thread)
2. rpm -Uvh qmail-toaster-1.03-1.3.21.src.rpm
3. bunzip2 /usr/src/redhat/SOURCES/qmailtoaster-1.3.2.patch.bz2
4. vi /usr/src/redhat/SOURCES/qmailtoaster-1.3.2.patch
5. Search for “/* #define CHKUSER_DISABLE_VARIABLE” and remove the comments, save
6. Search for “/* #define CHKUSER_ENABLE_USERS_EXTENSIONS” and remove the comments to re-enable dash-aliasing
7. bzip2 /usr/src/redhat/SOURCES/qmailtoaster-1.3.2.patch
8. cd /usr/src/redhat/SPECS
9. rpmbuild -bb –with cnt50 –target i686 qmail-toaster.spec
10. rpm -Uvh /usr/src/redhat/RPMS/i686/qmail-toaster-1.3.2.rpm

Note, I’m running on x64 CentOS 5. For some reason I had to specify the target as i686 as without the target it wanted to generate an i386 binary while everything else is i686. I didn’t know if that would cause problems so I specified the target explicitly.

For any IP address we want to bypass the CHKUSER checks, we simply add an entry to /etc/tcprules.d/tcp.smtp with a RELAYCLIENT variable. An entry might look like:

192.168.0.1:allow,RELAYCLIENT=""

Finally, the RPM update for qmail-toaster overrides your SSL certificates for SMTP. To restore my real certificates, I had to perform the following steps (lifted from coregilmore.com):

# cp /etc/httpd/certs/mail.msr.com.pem /var/qmail/control/servercert.pem
# ln -s /var/qmail/control/servercert.pem /var/qmail/control/clientcert.pem
# chown -h qmaild:root /var/qmail/control/clientcert.pem
# chmod 400 /var/qmail/control/servercert.pem
# qmailctl restart

mail.msr.com.pem is simply a single concatenated file containing your key, certificate and any intermediate certificates (required for registrars like Go Daddy for example):

# cat /path/to/ssl_cert.key /path/to/ssl_cert.crt /path/to/gd_intermediate_bundle.crt > /var/qmail/control/servercert.pem

Updated Mailfilter Script

We also expanded our mailfilter script to be a little bit smarter and handle DSPAM better. Here’s the latest:

SHELL="/bin/bash"
import EXT
import HOST

VUSER=`echo ${EXT%-*}`
export VUSER
USERHOME=`/mail/bin/vuserinfo -d $VUSER@$HOST`
export USERHOME

MARKTIME=`date +%s.%N`
UNIQUE=`date +%N`

logfile "/var/log/maildrop/mailfilter.log"

# Test for the existance of the Junk directory.  Create it if it doesn't exist
VERBOSE=1
log "$VUSER@$HOST[$UNIQUE] : Start user filter"

# make sure the Junk folder exists and is subscribed too
`test -d $USERHOME/Maildir/.Junk`

if ( $RETURNCODE == 1 )
{
    `/usr/bin/maildirmake -f Junk $USERHOME/Maildir`
    `echo INBOX.Junk >> $USERHOME/Maildir/courierimapsubscribed`
}

# make sure the retrain folder exists
`test -d $USERHOME/Maildir/.JunkRetrain`

if ( $RETURNCODE == 1 )
{
    `/usr/bin/maildirmake -f JunkRetrain $USERHOME/Maildir`
    `echo INBOX.JunkRetrain >> $USERHOME/Maildir/courierimapsubscribed`
}

# now analyze the SA header
if (/^X-Spam-Status: Yes/)
{
        log "$VUSER@$HOST[$UNIQUE] : SpamAssassin found SPAM"
        to "$USERHOME/Maildir/.Junk"
}
else
{
        # spamassassin thinks its ham, which means nothing, because SA sucks
        # so now we check with dspam

        # warning: exception must have one space and then the curly bracket or else syntax errors are generatead
        exception {
                xfilter "/usr/local/bin/dspam --deliver=innocent,spam --stdout --user $USERHOME"
        }

        if (/^X-DSPAM-Result: Spam/)
        {
                # dspam says spam
                log "$VUSER@$HOST[$UNIQUE] : DSPAM found SPAM"
                to "$USERHOME/Maildir/.Junk"
        }
        else
        {
                # ham in both places, deliver
                log "$VUSER@$HOST[$UNIQUE] : DSPAM found HAM (returncode = $RETURNCODE)"
                if (/^X-DSPAM-Result: Innocent/)
                {
                        log "$VUSER@$HOST[$UNIQUE] : HAM includes X-DSPAM-Result: Innocent"
                }
                else
                {
                        if (/^X-DSPAM-Result: Whitelisted/)
                        {
                                log "$VUSER@$HOST[$UNIQUE] : HAM includes X-DSPAM-Result: Whitelisted"
                        }
                        else
                        {
                                log "$VUSER@$HOST[$UNIQUE] : HAM does NOT include X-DSPAM-Result: Innocent/Whitelisted"
                        }
                }
                to "$USERHOME/Maildir/"
        }
}

Since we’re dropping SpamAssassin, this file could be streamlined to be DSPAM specific but it doesn’t hurt to leave it in the event we decide to re-enable it at a later date. Simply save the script somewhere and enable it via a .qmail file in a user directory with the following:

|preline /usr/bin/maildrop /path/to/mailfilter-to-spam-plus-dspam

I had been considering outsourcing everything – move incoming email to Google Apps and use someone like SMTP.com or SendGrid for outbound but there’s a few wrinkles like mailing lists and customer-relayed email. We ultimately decided to roll a new machine but given how quickly email deliverability evolves, we needed to get up to date with the various authentication and verification technologies that help get emails into the inbox. That meant getting waist-deep in SPF, Sender-ID, DomainKeys and DKIM. (See current DKIM and SPF deployment rates – 23% and 51% globally as of this writing)

This isn’t a step-by-step guide; writing one would simply repeat what other people have already done a good job of documenting. What this is, is an overview that ties together missing pieces and endless Google searches to move from concept to execution with one full set of examples.

The Moving Pieces

If you’re new to all this, here’s how these technologies work briefly: Sender ID is basically a Microsoft version of Sender Policy Framework (SPF). If you get SPF to work, Sender ID is backwards compatible and will verify successfully. DKIM is almost the same thing to DomainKeys but it adds Author Domain Signing Practices (ADSP) as an optional component that specifies how strict you are about signing messages.

SPF and Sender ID rely on specially crafted DNS records that identify which mail servers are allowed to send mail for your domain. By contrast, DomainKeys/DKIM take pieces of your message, cryptographically hash them with a key (part of which only your server knows) and embed the signature as a header which is verified using the other half of your key (which is public) that is stored in a specially crafted DNS record for your domain or mail server. You must have the ability to create DNS TXT records or none of these will work for you. Here’s a good article with a more details on how it all works.

The idea is to eliminate any joe spammer from using an arbitrary From address in spam, phishing and other email attacks. None of these approaches are perfect but they are used by large ISPs like Hotmail, Gmail and Yahoo so if you want your email to reach those inboxes, you need to play along.

QmailToaster

Step one, set up a QmailToaster. Our current server is (older) Qmail + Vpopmail + Courier based and it has worked well so we stuck with what we know. Not to mention, Jake Vickers and the QmailToaster team have removed all of the pain of setting it up so now you just speed along to the part where it works well. Other than getting the needed dependencies on my CentOS/RHEL box, the installation was a piece of cake using the qtp-newmodel approach.

• http://wiki.qmailtoaster.com/index.php/CentOS_5_QmailToaster_Install – the CentOS/RHEL 5 Toaster install gives the quick and dirty of adding DomainKeys support
• http://wiki.qmailtoaster.com/index.php/Domainkeys – detailed directions for DomainKeys support
• http://wiki.qmailtoaster.com/index.php/How_to_Setup_DKIM_with_Qmail_Toaster – Add DKIM support with a new qmail-queue

After you install the DKIM package from above on RHEL/CentOS 5, you would see this error in your qmail logs:

@400000004d7d3bc00da74aec delivery 21: success: ould_not_find_ParserDetails.ini_in_/usr/lib/perl5/vendor_perl/5.8.8/XML/SAX/rUser_and_password_not_set,_continuing_without_authentication./<check-auth@verifier.port25.com>_96.244.219.19_accepted_message./Remote_host_said:_250_2.6.0_message_received/

Basically, something is borked in the CPAN-installed SAX package. You can fix it by pre-emptively running (thanks to centosfiles for the fix):

perl -MXML::SAX -e "XML::SAX->add_parser(q(XML::SAX::PurePerl))->save_parsers()"

At this point you have a mail server capable of handling SPF, Sender ID, DomainKeys and DKIM. Now you need to enable it in your configuration and your DNS.

SPF/Sender ID

Unlike DomainKeys and DKIM which require public/private keys and some computational effort when delivering email, SPF and Sender ID are exclusively a DNS configuration option.

• http://wiki.qmailtoaster.com/index.php/SPF – there technically isn’t anything to set up with the QmailToaster since your outbound configuration is all DNS. However, you can configure how to handle incoming email that doesn’t pass using /var/qmail/control/spfbehavior.
• Qmail SPF – The underlying implementation of SPF for qmail
• OpenSPF.org’s Record Wizard – generate an SPF record for your domain to be placed in a DNS TXT record

Once generated, take the TXT record and put it into your DNS files. If you’re using djbdns, a simple record with one mail server would look something like:

'sample.com:v=spf1 ip4:your.ip.add.ress ~all:3600

The ~all says to softfail messages that don’t verify. If you are sure you have listed all of your possible outbound MTAs, then you can set it to -all which will tell remote servers to kill emails that don’t pass.

DomainKeys/DKIM

Once installed, you enable incoming verification and outgoing signing through the tcp.smtp file stored in /etc/tcprules.d:

• http://wiki.qmailtoaster.com/index.php/Domainkeys#Outgoing – using DKSIGN for outgoing messages and DKQUEUE/DKVERIFY for incoming messages. Has a flag-by-flag breakdown of how to configure DKVERIFY for how to respond to incoming mail that passes or fails verification.
• http://wiki.qmailtoaster.com/index.php/Tcp.smtp – a complete breakdown of the tcp.smtp file that includes DKSIGN, DKVERIFY and DKQUEUE breakdown
• HOWTO define DKIM/ADSP RRs – from the “Pro DNS and BIND” book, this page does a phenomenal job of explaining the components of the DomainKeys/DKIM records with examples. Also explains selectors clearly.

For DKVERIFY, the default is “DEGIJKfh”. If you want to permanently reject incoming mail that has a DomainKey signature but fails to verify, add a “B” to the list: “BDEGIJKfh”.

Outbound

To sign an outbound message for DomainKeys/DKIM, you need to generate a public/private key pair. Part of it you keep secret and sign your outgoing messages and the other half you publish in your DNS so other mail servers can verify it’s authentic. Here’s what the docs say to do:

# cd /var/qmail/control/domainkeys
# mkdir yourdomain.com
# cd yourdomain.com
# dknewkey private > public.txt
# chmod 440 private
# cd ..
# chown -R root:vchkpw yourdomain.com

The word “private” above is going to be our selector. It’s 100% arbitrary but must be used consistently. QmailToaster defaults to the word “private”. In your DNS, you need a TXT record that looks something like:

private._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ . . .

The rest of the record comes out of the file we just created with dknewkey in /var/qmail/control/domainkeys/yourdomain.com/public.txt. In my last Qmail install, I used “default” for my selector, so my DNS entries look like:

default._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ . . .

This selector, be it “private”, “default” or something else, MUST match what you use to generate your public/private key using dknewkey and it must also match what you use in your /etc/tcprules.d/tcp.smtp rules where you specify when to sign outgoing messages:

# example record using QmailToaster defaults
127.:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/private"

# since I used 'default' as my selector, this is what my file looks like:
127.:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/default"

Crypto Warning

dknewkey creates a (relatively) short 384-bit public/private keypair. There are people who don’t think that’s good enough and they’re probably right. You can edit /usr/bin/dknewkey to change it to something more secure like 512 or (better yet) 1024 bits.

A quick recap with how this selector is used throughout your Qmail and DNS configuration:

1. Generate the public/private key: dknewkey my-selector-name > public.txt; this leaves behind a file like /var/qmail/control/domainkeys/yourdomain.com/my-selector-name.
2. Create the DNS TXT record: my-selector-name._domainkey.yourdomain.com:k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQ… (using the contents of the public.txt file dknewkey creates)
3. In tcp.smtp, use a DKSIGN environment variable: DKSIGN=”/var/qmail/control/domainkeys/%/my-selector-name”

Testing

Once you have it set up and your DNS has propagated, there are a number of online services and email reflectors that will verify your configuration and report back to you:

Web Tools

• SourceForge DomainKeys Policy Checker
• Scott Kitterman’s SPF record checker – test your SPF record from the web
• DKIM DNS Record Wizard – based on RFC 4871, helps construct the TXT record for your DNS for DKIM signing
• SendMail ADSP wizard and DKIM verifier – helps build an (optional) ADSP TXT record and can check your DKIM DNS entries

Email Reflectors

Unlike the above services which try to see if things look OK, these are email addresses you can send a test message to that will actually tell you how your email performed:

• check-auth@verifier.port25.com – the port25 reflector sends you an email report with a summary and breakdown by type. This is the best one in my opinion.
• spf-test@openspf.org – this message will be rejected but the rejection will say what happened to the SPF test. You must tail the logs (like /var/log/qmail/smtp/current) to see the results.
• autorespond+dkim@dk.elandsys.com – Eland Systems offers a DKIM reflector along with some notes
• MyIpTest.com Tester – tests DK, DKIM and SPF. Web page gives you a random email address to send to and then you can load your results after sending a test message.
• http://www.brandonchecketts.com/emailtest.php – similar to above, send an email to a random address provided on the web page and then you can check results.

Port25’s reflector report should look like this when everything is working:

Summary of Results
==========================================================
SPF check: pass
DomainKeys check: pass
DKIM check: pass
Sender-ID check: pass
SpamAssassin check: ham

Microsoft will let you “register” your Sender ID settings with them. Not sure how important this is, but it can’t hurt to let them know at https://support.msn.com/eform.aspx?productKey=senderid&page=support_senderid_options_form_byemail.

Email Services

Better than a reflector is to see how a real mail service will treat your message. Once you think you’re configured correctly, try sending an email to Gmail and Yahoo. Then you can view the headers for each to see how your SPF and DK records were interpreted.

Gmail:

Yahoo:

Final Thoughts

Putting it all together, here’s how my setup looks (using the selector ‘default’):

Public/private key pairs on file system

[root@mail domainkeys]# ls -al /var/qmail/control/domainkeys/sample.com/
total 32
drwxr-xr-x 2 qmailq root  4096 Mar 13 13:46 .
drwxr-xr-x 7 root   qmail 4096 Mar 13 13:46 ..
-r--r----- 1 qmailq root   396 Mar 13 13:46 default
-rw-r--r-- 1 qmailq root   142 Mar 13 13:46 public.txt

DKIM signconf.xml

On the file system in /var/qmail/control/dkim/signconf.xml:

<dkimsign>
  <global algorithm="rsa-sha1" domain="/var/qmail/control/me" keyfile="/var/qmail/control/dkim/global.key" method="simple" selector="dkim1">
    <types id="dkim" />
  </global>
</dkimsign>

You could override the use of /var/qmail/control/me or change the selector from ‘dkim1′ if needed.

Alternative note: It is also possible to sign your DomainKeys using the DKIM package. You can change signconf.xml to have a per-domain record with DK and then abandon DKSIGN altogether (this is the route I’ve gone):

<sample.com domain="sample.com" selector="default">
  <types id="dkim" />
  <types id="domainkey" method="nofws" />
</sample.com>

tcp.smtp

# localhost relaying
127.:allow,RELAYCLIENT="",QMAILQUEUE="/var/qmail/bin/simscan",DKVERIFY="DEGIJKfh",DKQUEUE="/var/qmail/bin/qmail-queue.orig",DKSIGN="/var/qmail/control/domainkeys/%/default"
# external web box
69.36.226.12:allow,RELAYCLIENT="",DKSIGN="/var/qmail/control/domainkeys/%/default"
# everyone else that connects gets the full cavity check
:allow,BADMIMETYPE="",BADLOADERTYPE="M",CHKUSER_RCPTLIMIT="50",CHKUSER_WRONGRCPTLIMIT="3",QMAILQUEUE="/var/qmail/bin/simscan",DKVERIFY="DEGIJKfh",DKQUEUE="/var/qmail/bin/qmail-queue.orig",DKSIGN="/var/qmail/control/domainkeys/%/default"

DNS TXT Records

Using djbdns:

'sample.com:v=spf1 ip4\07269.36.226.12 include\072emailcenterpro.com ~all:3600
'_domainkey.sample.com.:o=-;:3600
'default._domainkey.sample.com.:k=rsa; p=MHwwDQYJKoZIhvcNAQEBBQADawAwaAJhAPBdKl9mx3KIjLwGfq83LNzUN4aDWc4t3PVTnc0d+duI9HVD+UG+i1suifV6obDzo4ugHji6EH+zei39Cch9vTXcpVWkaFCEmVu0AXjY/98WIjYb5Hh5+SYQFyQkz0Mq0wIDAQAB;:3600
'dkim1._domainkey.mail.sample.com.:v=DKIM1; k=rsa; p=MEwwDQYJKoZIhvcNAQEBBQADOwAwOAIxAK+vhWHoPv+3DhM1u3MaHk7AayBa+CdNnjPHhg3/3Nv7hEIYUbOfKSUqNxDtjJkgcQIDAQAB;:3600
'_adsp._domainkey.mail.sample.com:dkim=all:3600
'_adsp._domainkey.sample.com:dkim=all:3600

First line is my SPF/Sender ID record. It also uses an include to allow a third party (in this case, EmailCenterPro) to send mail on our behalf.

The second line says that we sign all outgoing messages with DomainKeys/DKIM (o=-).

Third line is for DomainKeys using our selector ‘default’ (which correspondingly shows up in tcp.smtp in DKSIGN and on the filesystem as /var/qmail/control/domainkeys/sample.com/default).

The fourth line is the DKIM record which uses the global.key and public.txt stored in /var/qmail/control/dkim. Why is it dkim1._domainkey? Because that’s the arbitrary selector specified in the signconf.xml file as listed above. Again, totally arbitrary, could be ‘private’ or ‘default’ or something else, but best to have it different from your DomainKeys. You might also notice that it is mail.sample.com instead of sample.com? By default it must match /var/qmail/control/me file because that’s how the message will be signed by the QmailToaster DKIM package.

The fifth and sixth line are ADSP records that say we sign all outbound messages with DKIM. Some hosts appear to be checking for it without the mail subdomain so I’m currently running both. I don’t think it should be strictly required but at this time seems to cover the bases.

The 3600 on every line is the TTL, meaning these records expire after 1 hour.

Quarantining Spam in IMAP Folder

Before upgrading, we used DSPAM as our spam filter of choice. In a word, it’s awesome. It quarantined mail so effectively that I eventually stopped checking the quarantine until it became so large I could no longer view or delete it from the web interface. The secret sauce in my setup, which involves Mozilla Thunderbird was to use Thunderbird’s built-in “Junk” folder as a feedback loop for DSPAM. For some reason, Thunderbird is really good at catching certain kinds of spam that DSPAM was not and, if a spam did get through, I can select it and press “J” which trains Thunderbird and moves the message to my Junk folder. The server processed those Junk folders nightly, feeding them to DSPAM to further increase its accuracy.

I like not downloading all of the assumed junk mail to my phone or, worse yet, on a slow net connection while traveling somewhere around the world. Unfortunately, QmailToaster’s SpamAssassin approach is to flag but leave it in your inbox. No bueno. Luckily we can use some of the software already installed and a very short script to replicate the previous behavior.

Create generic mailfilter

If you accept users will use a “Junk” folder (or “Spam” or whatever you want to call it), then you can create a single maildrop mailfilter script in ~vpopmail/etc/mailfilter-to-spam:

SHELL="/bin/bash"
import EXT
import HOST

VUSER=`echo ${EXT%-*}`
export VUSER
USERHOME=`/mail/bin/vuserinfo -d $VUSER@$HOST`
export USERHOME

logfile "/var/log/maildrop/mailfilter.log"

# Test for the existance of the Junk directory.  Create it if it doesn't exist
#VERBOSE=9
#log "testing for $VUSER@$HOST in $USERHOME"

if ( /^X-Spam-Status: Yes/ )
{
        `test -d $USERHOME/Maildir/.Junk`

        if ( $RETURNCODE == 1 )
        {
            `/usr/bin/maildirmake -f Junk $USERHOME/Maildir`
            `echo INBOX.Junk >> $USERHOME/Maildir/courierimapsubscribed`
        }

        to "$USERHOME/Maildir/.Junk"
}
else
{
        to "$USERHOME/Maildir/"
}

Now create a ~vpopmail/domains/sample.com/username/.qmail file for those users who want their spam quarantined in a “Junk” folder rather than just tagged:

|preline /usr/bin/maildrop ~vpopmail/etc/mailfilter-to-spam

Done! All messages that meet your SpamAssassin minimum score (default of 5.0) will be quarantined in the “Junk” folder for review from SquirrelMail or your IMAP client. You can process these folders using a script like the following (warning, not tested in production yet!):

#!/bin/bash
cd ~vpopmail/domains

# find all Junk folders
for ii in `find . -type d -name '.Junk'`; do

  # find all mail 5 days or older
  for jj in `find $ii/cur/ -type f -mtime +4`; do

    # learn from it and delete it
    echo "processing $jj"
    /usr/bin/sa-learn --spam --no-sync $jj
    rm -f $jj

  done;

done;

# now synchronize the overall SA database
/usr/bin/sa-learn --sync

The argument “-mtime +4″ says look for files that are 5 days or older. You could adjust that to +1 for 48 hours or “-mmin +60″ for 60 minutes or older depending on your policies. Schedule this to run as a cronjob nightly and you’ll be provided an ongoing set of spam to learn from.

Hope these links and notes help someone else in the future!

Now, to the script:

#!/bin/bash

# init vars
BACKUP_ROOT=/dblogs/backups
S3CMD=/usr/bin/s3cmd
S3DIR=s3://your-bucket-name/your-folder-name

# take the last X backups and make sure they are on S3 (but not older ones)
BACKUP_COUNT=5
BACKUP_FILELIST=/tmp/.pgbackup_s3cmd_filelist

# first list out the files we want to sync up
ls -1 $BACKUP_ROOT | tail -$BACKUP_COUNT > $BACKUP_FILELIST

# now sync them; requires trailing slash to sync directories
$S3CMD sync --progress --delete-removed --acl-private --exclude '*' --include-from $BACKUP_FILELIST $BACKUP_ROOT $S3DIR/

# clean up
rm -f $BACKUP_FILELIST

Setup s3cmd after installing it by running s3cmd --configure with your Amazon credentials handy. I also use s3fox, a Firefox add-on, as another way of quickly accessing S3 with a GUI.

This script maintains only the last 5 backups on S3 and it deletes what would be the 6th backup each night via cron. You could adjust the number to keep with the BACKUP_COUNT parameter. I’m personally using this to backup a PostgreSQL server but you can adjust it to back up any type of directory with files in it.

Update 2/17 – realized the default behavior is not to delete files on the remote host that are not included locally. I updated the above to include the –delete-removed argument and now it deletes the oldest file as expected.

01/10 13:03:28 user MessageBrokerServlet: init
**** 90 seconds of "working" ****
01/10 13:05:13 user CFFormGateway: init

01/10 12:32:58 Information [scheduler-12] - Starting debugging...
01/10 12:32:58 Information [scheduler-12] - Starting sql...
****  almost 2 minutes here ****
01/10 12:34:46 Information [scheduler-12] - Pool Manager Started
01/10 12:34:46 Information [scheduler-12] - Starting mail... 

This simply would not do. Not only was it slow, it was unpredictable, sometimes taking 90-115 seconds and other times as long as 224 seconds. We generally restart our CF servers when we do deployments and the sluggish startup meant more queued requests and timeouts as we come back from an update. No bueno.

I turned to my friends on the CFGURU list hoping someone had the answer. The always helpful Dan Switzer pointed me to a cfaussie thread which was Ubuntu specific but had my same behavior:

With the startup you can see that cranking up the Message Broker Servlet (ie Livecycle/Blaze) is taking almost a minute, which I find is usually due to some network ports used by LCDS still being open from the last time ColdFusion was run.

The good news was that they were experiencing the exact same problem I was. The bad news is that I have nothing to do with Blaze or LiveCycle so I wasn’t sure this was related. We are also running RHEL/CentOS but the Ubuntu connection proved helpful later in my research.

Anytime I see hanging processes in ColdFusion, I generate a complete JVM stack trace. Unfortunately, I’m quite bad at reading them. I identified a BLOCKED process but Daryl Banttari took a look and set me straight:

This is the thread that's your root problem:

"scheduler-10" prio=10 tid=0x00002aaaac28c800 nid=0x5edd runnable [0x00000000441de000]
   java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:199)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00002b4fc75a5018> (a java.io.BufferedInputStream)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0x00002b4fc75a4ce0> (a java.io.BufferedInputStream)
at sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte(SeedGenerator.java:453)
at sun.security.provider.SeedGenerator.getSeedBytes(SeedGenerator.java:123)
at sun.security.provider.SeedGenerator.generateSeed(SeedGenerator.java:118)
at sun.security.provider.SecureRandom.engineGenerateSeed(SecureRandom.java:114)
at java.security.SecureRandom.generateSeed(SecureRandom.java:495)
at com.rsa.jsafe.provider.c7.engineGenerateSeed(Unknown Source)
at java.security.SecureRandom.generateSeed(SecureRandom.java:495)
at java.security.SecureRandom.getSeed(SecureRandom.java:482)
at com.rsa.jsafe.crypto.au.g(Unknown Source)
at com.rsa.jsafe.crypto.au.a(Unknown Source)
at com.rsa.jsafe.provider.JSA_FIPS186PRNGXChangeNoticeGeneral.engineNextBytes(Unknown Source)
at java.security.SecureRandom.nextBytes(SecureRandom.java:433)
- locked <0x00002b4fdca32370> (a java.security.SecureRandom)
at java.security.SecureRandom.next(SecureRandom.java:455)
at java.util.Random.nextInt(Random.java:253)
at flex.messaging.util.UUIDUtils.createUUID(UUIDUtils.java:110)
at flex.messaging.util.UUIDUtils.createUUID(UUIDUtils.java:97)
at flex.messaging.log.AbstractTarget.<init>(AbstractTarget.java:52)
at flex.messaging.log.LineFormattedTarget.<init>(LineFormattedTarget.java:79)
at flex.messaging.log.ConsoleTarget.<init>(ConsoleTarget.java:34)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at java.lang.Class.newInstance0(Class.java:355)
at java.lang.Class.newInstance(Class.java:308)
at flex.messaging.config.MessagingConfiguration.createLogger(MessagingConfiguration.java:298)
at flex.messaging.MessageBrokerServlet.init(MessageBrokerServlet.java:114)
at coldfusion.flex.ColdFusionMessageBrokerServlet.init(ColdFusionMessageBrokerServlet.java:24)
at coldfusion.bootstrap.ClassloaderHelper.initServletClass(ClassloaderHelper.java:94)
at coldfusion.bootstrap.BootstrapServlet.init(BootstrapServlet.java:59)
at jrun.servlet.WebApplicationService.loadServlet(WebApplicationService.java:1213)
- locked <0x00002b4fdbc4e5c8> (a java.lang.Object)
at jrun.servlet.WebApplicationService.preloadServlets(WebApplicationService.java:793)
at jrun.servlet.WebApplicationService.postStart(WebApplicationService.java:295)
at jrun.ea.EnterpriseApplication.start(EnterpriseApplication.java:203)
at jrun.deployment.DeployerService.initModules(DeployerService.java:708)
at jrun.deployment.DeployerService.createWatchedDeployment(DeployerService.java:243)
at jrun.deployment.DeployerService.deploy(DeployerService.java:428)
- locked <0x00002b4ed0c3f140> (a java.util.HashMap)
at jrun.deployment.DeployerService.checkWatchedDirectories(DeployerService.java:179)
at jrun.deployment.DeployerService.run(DeployerService.java:889)
- locked <0x00002b4ed0c3f140> (a java.util.HashMap)
at jrunx.scheduler.SchedulerService.invokeRunnable(SchedulerService.java:230)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Looks like it's stuck trying to read from /dev/random while trying to create a UUID?

The key was that Java was waiting on input to the secure number generator code. Thanks to Daryl’s stack trace reading kung fu, I had something to Google. On Linux, /dev/random is a “blocking” number generator meaning if it doesn’t have enough random data to provide, it will simply wait until it does. Here’s some background on /dev/random. Keyboard and mouse input as well as disk activity can generate the randomness, or entropy, needed but perhaps not fast enough for particular applications. A lack of random data would force the JVM and ColdFusion to wait, for eternity if necessary, until chaos caught up with /etc/init.d/coldfusion8multi.

It turns out a lot of applications experience this problem: any that use random number generators. I found frustrated users of Apache, SSL, Livecycle/Blaze DS and a myriad of other applications as my search terms refined.

Verify the culprit

Before you try the changes below, verify that this is indeed your problem. It’s easy! First, start by checking how much entropy you have:

cat /proc/sys/kernel/random/entropy_avail

This kernel facility in /proc has been available since 2.3.16 so it should work on any reasonably recent Linux. A full pool should be 4096 (check your poolsize config in /proc/sys/kernel/random/poolsize). On my normal servers, I had values around 3500 but this particular server was 150. Stop ColdFusion (or Apache or LCDS or whatever app is giving you fits) and run the following command before starting CF again:

watch –interval 0.1 cat /proc/sys/kernel/random/entropy_avail

Now start ColdFusion and what you may see is as it gets to MessageBrokerServlet, the available entropy drops to near-zero, climbs back up to around 50-70, then drops to zero and repeats like a stairmaster until it completes and CF can resume at a normal pace.

Make random data non-randomly appear

Now we know roughly what the problem is: there ain’t enough random data. How do we make it so ColdFusion and JRun can start up faster? We need to ensure that there is a constantly available source of entropy so the JVM can seed its secure number generator quickly.

Armed with new search terms for Java’s SecureRandom, I came across a StackOverflow thread which offered a JVM argument to switch from the blocking /dev/random to the less-secure-but-non-blocking /dev/urandom:

-Djava.security.egd=file:/dev/./urandom

I appended that onto the end of java.args in /opt/jrun4/bin/jvm.config and the server started up in 12 seconds! I felt a little bit of hair on my head grow back at that moment.

Solving for Production

At this stage I’ve identified the culprit but I had uneasy feelings about using /dev/urandom in production where evil doers are constantly trying to do evil things to our server, application and users. We don’t want an insecure random number generator.

Following the Ubuntu connection from Stack Overflow, this post on slow Apache starts described the same symptoms and the use of rngd (a random number generator daemon) to populate the entropy pool. Rngd is included in rng-tools or rng-utils depending on your distro. On RHEL 5.5, it was already installed under /sbin.

The trick is to feed rngd data from the kinda-insecure /dev/urandom and let it figure out what is secure enough to pass to /dev/random with a FIPS-140 test. You can run rngd two ways. If you want to populate /dev/random once, right now, run it in the foreground:

/sbin/rngd -r /dev/urandom -o /dev/random -f -t 5

Ctrl-C out of it after 15 seconds and check entropy_avail – you should find it’s up around 4000. Very nice! Now, how about running it as a daemon so you don’t have to ever worry if ColdFusion has enough to start quickly? I added this to my /etc/rc3.d/S99local file so it will automatically start every time the server is started:

/sbin/rngd -r /dev/urandom -o /dev/random -t 5
 

Fire that up, restart ColdFusion, crack a beer and ride off into the sunset. Another ridiculous unexpected behavior solved by yours truly.

Interesting side note: Once upon a time, Intel included hardware RNGs in their chip sets that would be exposed as /dev/hw_random. If you had one of those, you could use it as the parameter for -r. Unfortunately that practice died out in the 800 series chips and are no longer available. You can still buy external, cryptographically secure hardware RNGs that can be used for rngd but the above seems to be “good enough” according to the tubes.

If you want to go nuts, you can use a radio tuned to static as an audio input to provide the source data. Unless your a crypto-nerd, this thread about RNG sources will make your eyes bleed.

Resources

EGD – entropy gathering daemon is a userspace substitute for /dev/random, written in perl. It could be used with the java.security.egd parameter above in lieu of /dev/random.

http://whatan00b.com/slow-apache-starts-on-ubuntu Llinked to from the StackOverflow thread on the Java SecureRandom slowness, explained entropy and how to monitor it.

http://blog.sbf5.com/?p=50 Pointed me in the right direction for using rngd to generate entropy for /dev/random

Increasing entropy pool on RHEL/Fedora without keyboard or mouse Another source to monitoring entropy and using rngd that turned up in Google as I got more specific with my searches.

Wikipedia on Hardware RNGs An overview of ways to source random data

Did you know? /proc/sys/kernel/random/uuid contains a fresh Version 4 UUID.

I drove for Team Bimmerworld in an early 1990s BMW M3 and was in the car at night when the race was temporarily stopped due to thick valley fog. The changing visibility from lap to lap was challenging but fun and all of the drama was captured on a ChaseCam. Unfortunately the memory card disappeared with BMW E30 rally-star Bill Caswell and we never got more than halfway in exchanging the data. The main problem? Bill is a non-stop motorsport animal always on the go and the file was 13GB.

At one point Bill managed to get the file uploaded to getdropbox.com but even installing their desktop client, I was unable to make it more than about 500MB at 5k/sec before timeouts or other network shenanigans would restart the process. Frustrating!

That was twelve months ago. This week I was cleaning out some old emails and came across the link to the file and, for fun, tried clicking it. Hey, it’s still there! And still 13GB.

On a whim I logged in to my web server to see what I could do with the venerable command-line utility wget. Turns out wget has a few tricks up its sleeves and some 60 hours later I’ve managed to download 5GB of the video. Here’s the magic in the command line:

wget --continue --progress=dot:mega --tries=0 <URL>

I was unaware of the continue option before which tells wget to try and restart any downloads where they left off. The progress option indicates 3MB per line of dots rather than 384k; appropriate for a file of this size. And finally, tries=0 means keep trying forever regardless of how many times the connection fails. Here’s what a failure looks like:

2010-11-26 10:25:48 (38.4 KB/s) - Connection closed at byte 5314183343. Retrying.

--2010-11-26 10:25:58--  (try:20)  http://dl.dropbox.com/u/PRIVATE-URL
Connecting to dl.dropbox.com|75.101.129.115|:80... connected.
HTTP request sent, awaiting response... 206 PARTIAL CONTENT
Length: 14425877949 (13G), 9111694606 (8.5G) remaining [video/quicktime]
Saving to: `2009 25 hours of thunderhill reel 10.mov'

         [ skipping 5188608K ]
5188608K ,,,,,,,, ,,,,,,,, ........ ........ ........ ........ 36% 61.7K 2d0h
5191680K ........ ........ ........ ........ ........ ........ 36% 38.1K 2d2h

Success! This is the 20th restart for 5GB averaging 500GB between connection failures. Thanks to these few little tweaks it should keep trying until it finishes and then I’ll finally have my footage!

For Linux, HP has a tool available called hpacucli (HP Array Configuration Utility for Linux) for interrogating HP/Compaq array controllers (SmartArray 5i, 6i, whatever) from the command line. Before you can install the RPM (on CentOS/Redhat), you will need to first install a compatibility library:

yum install compat-libstdc++-296
rpm -Uvh hpacucli-8.0-14.noarch.rpm 

Then I put this snippet into a new file /etc/cron.hourly/raidstatus:

#!/bin/sh
/opt/compaq/hpacucli/bld/hpacucli ctrl all show config | egrep -i "(fail|error|offline|rebuild|ignoring|degraded|skipping|nok)"

The command /opt/compaq/hpacucli/bld/hpacucli ctrl all show config normally generates something like this (from our development database server):

Smart Array XXXXXXX in Slot 0      ()

   array A (Parallel SCSI, Unused Space: 0 MB)

      logicaldrive 1 (33.9 GB, RAID 1+0, OK)

      physicaldrive 2:0   (port 2:id 0 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:1   (port 2:id 1 , Parallel SCSI, 36.4 GB, OK)

   array B (Parallel SCSI, Unused Space: 0 MB)

      logicaldrive 2 (67.8 GB, RAID 1+0, OK)

      physicaldrive 2:2   (port 2:id 2 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:3   (port 2:id 3 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:4   (port 2:id 4 , Parallel SCSI, 36.4 GB, OK)
      physicaldrive 2:5   (port 2:id 5 , Parallel SCSI, 36.4 GB, OK)

I believe you can reduce the grep to just “(fail|nok)” but I’m taking the conservative approach here. Change the permissions to 0700 and if you have SELinux running make sure the context is set properly.

If your array and controller are in fine shape, then this command will output nothing. If you have a dead drive, it will generate content which will cause cron to mail the root user about it. Bingo – time to go to the colo!

I have seen other people use “ctrl all show status” which generates:

Smart Array XXXXXXX in Slot 0
   Controller Status: OK
   Cache Status: OK
   Battery Status: OK

I prefer to query the config which looks at individual physical drives in addition to the status of the array. I have seen cases (just last week) where one dead drive in the array still lists the array status as OK (because, technically, it is OK, it’s just not optimal and may be pending major disaster!)

Update 5/25/2011 Had an error today, needed this code and corrected a few things. I fixed a typo for a missing quote in the raidstatus script above and added a link to the actual utility. For reference, the output from an error looks like this:

physicaldrive 2:2   (port 2:id 2 , Parallel SCSI, ??? GB, Failed)
physicaldrive 2:5   (port 2:id 5 , Parallel SCSI, 72.8 GB, Rebuilding, active spare)

Three weeks ago I grabbed the latest ActiveMQ 5.2.0 and I’ve been running it locally in development and production. I haven’t had a single queue failure since. There were over 100 bugs fixed in this release so I would strongly recommend upgrading if you’re using ActiveMQ with TransferSync to keep your cluster of ColdFusion servers synchronized.

I leave CFIDE and WEB-INF in the default location and create a symbolic link to it in the webroot of any vhost that needs access to it. You need access to it for the scripts and CSS for any AJAX stuff, so it may be important to do this. I like having CFIDE in a consistent, central location and adding links to it where needed, and I like having a fully-default CF install so that things are always the same.

I agree with his closing sentiment; the longer I administer servers the more I have become a fan of default installs and default settings. While getting it “just the way you like it” is great, it’s not a method that scales and it’s not easy to replicate unless you fully automate from the beginning.

Jared’s post made me want to share a quick tip I use in my Apache configuration files for the CFIDE directory. I work on both Windows and Linux so I try to avoid symbolic links which are difficult to manage in Subversion and on multiple platforms. Instead, I use a single-line Apache directive to accomplish the same thing in httpd.conf:

Alias /CFIDE /opt/jrun4/servers/cfusion/cfusion-ear/cfusion-war/CFIDE

Although the administrator has its own password, I like to further secure the ColdFusion administrator to prevent people from probing for vulnerabilities or attempting a DoS:

<Location /CFIDE/administrator>
Options +FollowSymLinks
Order deny,allow
Allow from all
AuthUserFile /opt/jrun4/servers/cfusion/cfusion-ear/cfusion-war/CFIDE/.htpasswd
AuthName "Administrators"
AuthType Basic
<Limit GET POST>
require valid-user
</Limit>
</Location>

I have a few other Apache tips and tricks if anyone finds these interesting?