Behind the Curtain at TCG

Upgrading to MySQL 5.5 on Ubuntu 10.10

I was recently asked to upgrade the MySQL install on a Ubuntu 10.10 system.  Normally these kinds of upgrades are simple affairs using built in tools like apt or yum, but in this case I was trying to go from MySQL 5.1 to version 5.5 and there were no pre-built packages to do it.

There were a few tutorials on-line that walked through different methods for the upgrade, but I couldn't find one that worked perfectly for me.  Maybe some packages we had upgraded or not upgraded were incompatible, or maybe I was just misreading some of the steps, or maybe there was something small missing from the tutorial. Whatever the reason, in each method I tried something always went wrong.  But on the plus side, by having a few resources on-line to show me how it worked for other people gave me all the info I needed to make it work on our system.

To help anyone else in this quest, I wanted to document what worked for me.

First, Backup!

The first thing you should do is backup all your data.  This includes the "mysql" schema and any other schemas you have setup. When following the directions you should replace "mydata" with the name of your actual databases, of course.

• cd /root
• mysqldump -p mysql > mysql.db.backup.sql
• mysqldump -p mydata > mydata.db.backup.sql

 

Install Pre-Requisites

Next there were a few tools that I needed to install so I could build MySQL from the source files:

• apt-get install cmake
• apt-get install libaio-dev

 

Uninstall the MySQL 5.1

Next, I uninstalled the existing MySQL 5.1

• /etc/init.d/mysql stop
• cp -R /var/lib/mysql /var/lib/mysql-5.1
• apt-get remove mysql-server-5.1
• apt-get autoremove
• apt-get remove mysql-client
• apt-get autoremove
• mv /var/lib/mysql /var/lib/mysql-5.1
• mv /etc/mysql /etc/mysql-5.1
• mv /usr/lib/mysql /usr/lib/mysql-5.1

 You may find that your data and configuration directories have been setup to live somewhere else.  If that's the case, use the proper locations rather than /var/lib/mysql or /usr/lib/mysql that worked for me.

Build and Install MySQL 5.5

Here are the build steps that I used.  If you don't have all the dev tools installed, you may need to apt-get the right bits. Hopefully the error messages will give you enough guidance to see what you need. This was the trickiest part for me and I did have to backtrack once or twice, so I hope that I didn't miss anything in my note taking.  If I did, I apologize in advance!

• wget -O mysql-5.5.21.tar.gz  http://dev.mysql.com/get/Downloads/MySQL-5.5/mysql-5.5.21.tar.gz/from/http://mysql.mirrors.pair.com/
• tar -xvzf mysql-5.5.21.tar.gz
• cd mysql-5.5.21.tar.gz
• cmake .
• make
• make install

• cd /usr/local/mysql/support-files/
• cp my-large.cnf /etc/my.cnf
• cd /usr/local/mysql/support-files/
• cp mysql.server /etc/init.d/mysql
• chmod +x /etc/init.d/mysql
• update-rc.d mysql defaults

 

Configure MySQL 5.5

You will need to configure your installation.  Here is what I did to get it setup the way we need for our projects. It's possible that you may need to do things a little differently:

edit /etc/my.cnf
   chagne the two socked lines to read:
       socket          = /var/run/mysqld/mysqld.sock
   add a datadir line after [mysqld]
       datadir         = /var/lib/mysql
  

Restore the Old Data

At this point I had a working installation with no data.  Hopefully you are in the same position.  The next step is to start mysql and restore whatever databases you backed up previously.  When following the directions you should replace "mydata" with the name of your actual databases, of course.

• mysqld --skip-grant-tables --user=mysql --datadir=/var/lib/mysql
• cd /root
• mysql -p mysql  < /root/mysql.db.backup.sql
• mysql -p mydata  < /root/mydata.db.backup.sql
• kill the running mysqld.  To do this you can find the pid of the process with "ps -eaf | grep mysqld" then do a kill -9 on the resulting pid if you have to.

Restart the Database and Apache

You should be ready to run the database properly now.  Since I am using this as a backing store for a web server, I'll restart my Apache instance as well:

• /etc/init.d/mysqld start
• /etc/init.d/apache restart

Give it a test and see if it worked.  I hope I was able to save you some time working this all out on your own.

Too Clever?

In many weakly-typed languages, such as Javascript and PHP, you might find code that takes some ambiguous value and coerces it into a straightforward boolean like so:

$boolean = (boolean) $mixed;

But what about this radically shorter alternative?

$boolean = !!$mixed;

Both rely on the typecasting features of the underlying language, but the latter does the job in with two characters, while the former takes six or more.

Is this second construct strange to you?  Not to me, not anymore.  I find it spare and quite quite clear, but perhaps I’m just used to it after using it so long I can’t even remember where I first saw it (or whether I possibly even invented discovered it on my own).

What other quirky coding do you find delightful... or infuriating?

The Beautiful Ternary

I am a fan of the ternary operator. Yes, it’s alien the first time you meet it, but wow, talk about getting things done! In a single line you can succinctly express a test and two options, making for truly compact yet clear assignments, for example:

var answer = ( condition ? optionOne : optionTwo );

But what if things are a little more complicated? Imagine alternatives involving really long snippets of code or nested options. These could be troubling with inline ternaries, leading to very long lines and complex statements difficult to apprehend at a glance.

But here comes convention to save the day! Consider this interesting use of spaces to visually lay out a ternary statement that would otherwise make for a very long line:

var product = (   determineProductClass( p1 )
                ? configureProductOne( p2, p3, p4, p5 )
                : configureProductTwo( p2, p3 ) );

Putting the three elements of the ternary on three separate lines is pretty common, but adding two spaces to the front of the first element and putting the operators at the front of the latter two lines is less common, I think. The effect, though, is to vertically align the three operands of the ternary, which to me makes the whole thing much more readable and easier to understand, especially if you think of the initial parenthesis and the extra spaces as a signal meaning “here comes a conditional.”

And here the same spacing convention goes wild to illustrate (literally!) a small set of nested options (which could go at least a few layers deeper easily without losing clarity):

var charge = (   product.isShippable()
               ? flatRateShippingCharge +
                 (   product.isOversized()
                   ? oversizedShippingSurcharge
                   : 0 ) // No surcharge for most products!
               : 0 ); // Digital product - no shipping!

Reminds me of the crazy comma first idea (but I’ll leave the tabs-versus-spaces elephant in the room). What do you think? Too weird... or actually helpful?

Every Project Needs a Cat ... or Why CMMI Works For Me

When the cat is away, the mice will play.

Earlier this month, TCG was re-confirmed as a CMMI Level 2 organization. Many people look at the CMMI documentation and think that it is a lot of overhead, but that's not the way I see it.  To me, CMMI is like keeping a cat in the kitchen.  

I'm not in any way saying that programmers, designers, testers, or managers are lazy mice that only work when the cat is away.  I'm lucky enough to work with people who are smart, motivated and want to do a great job.  In fact, everyone I work with not only wants to do a great job, but they know exactly what they need to do to make a project succeed in the long term.  

It's the "long term" bit that sometimes becomes a problem. Every project has deadlines and every team gets in a crunch now and again. What does a good development team do that situation? The best they can. But when you are down in the weeds you are lucky to see the upcoming mile stone.  You surely aren't thinking about the 5th or 6th milestone down the lane. Even a good team will sometimes deviate a bit from what they know are best practices if it means satisfying a client today.  You can always pay it back later, right?  Right?  Well...

My experience says that when things get tight, taking short cuts to meet today's goal just messes things up worse for tomorrow. Skipping a step you know to be good for the long term health of a project won't be good for the long term health of the project. Sure, that last statement seems obvious, but we've all been in meetings where someone says "well, since the deadline is this week I think we better just..."

So, back to the proverbial cat:  Having a defined and documented process that spells out the best process for your team is great. Having someone check up on folks to make sure they are following the process is even better. That's the cat right there. You will make the hard choice and do what is right because now you have to.  You can't sweep it under the rug.  

If you have stuck with me this far, you are surely thinking of that one time where you would have lost a $10 million project if you had to stick to the letter of every policy.  I agree, there are times when you have to bend a little to keep from snapping in half.  A good CMMI compatible policy will have enough flexibility to get you out of a jam.  It will typically involve presenting a good case for a waiver and getting sign off from "the management."  Not usually too hard if the alternative is wasting time/money or crashing a key project.  

So if you are looking at CMMI, don't see it as a busy work generator or a dusty binder of stuffy rules. It's not either of those things. CMMI is a framework that let's you define the process that is right for your team and then help you follow it. If anyone isn't following it, the cat will catch them sooner or later and fix the problem.

Web Accessibility Pays Off in Spades ... or Why You Shouldn't Count on JavaScript

Every web designer tries to make their sites as usable, understandable, and beautiful as possible.  Web accessibility is the practice of ensuring that you site is equally usable and understandable for all people, including the disabled.  It may not be obvious, but many of the web accessibility best practices will help you create a a site that is actually better for all users, not just those people we generally think of as disabled.  

Consider a very common disability that doesn't usually jump to mind when you say "disabled": color blindness.  About 7% of men and .4% of women in the united states have some form of color blindness. Because of this, it is not a great idea to use color alone to signal some special functionality on a web page. But "color sighted" users shouldn't have a problem, right?  Well, not every monitor is properly calibrated and on some low cost LCD monitors your users may have trouble distinguishing colors that are plainly different on your screen.  Even high contrast colors may not show up in black and white printouts of your site.

Setting aside color blindness, let's look at the much more serious case of total blindness.  In the early days of website development, i.e. the 1990's, vision impaired users employed screen reading software to translate web pages to spoken language.  These early readers did not support JavaScript.  Consequently, one component of web accessibility has traditionally been to make sure your site will work without JavaScript.  That has change recently though.  A 2010 survy of screen reader users shows that the vast majority of screen reader users have JavaScript enabled.  So, no need to worry about that anymore....right?  Well, not so fast.

If you rely heavily on JavaScript with no fallback, it is very easy for a small bug somewhere in your code to disable your entire site.  This very thing happened to Gawker Media in 2011 and put a number of top ranking web sites offline for a few hours.  JavaScript dependent sites can fail miserably leaving you with a blank page that isn't always easy to debug.  If you plan from the start to have a non-JavaScript fall back for all key functionality, you won't have this problem.

This week I spent some time going through NITRC.org to make sure we didn't have any obvious accessibility issues. During the investigation I did find some select lists using for sorting reports which required JavaScript to run properly. Users without JavaScript (for whatever reason) would not be able to sort of filter the reports.  Not a huge problem, but it is easily fixed:

<select name="xxx" onchange="this.form.submit();">
    <option>A</option>
    <option>B</option>
</select> 
<input type=submit value="go" class="hide-with-javascript" />
<script>$(".hide-with-javascript").css("display","none");</script>'

It is a very simple change and "regular" users who have JavaScript enabled won't even see the added button because the JQuery script snippet will hide it.

Another web accessibility best practice is providing "Alt" tags for images.  The Alt tag provides a text description of the image. Originally this was used so that screen readers could describe the image to a blind user. Now these Alt tags provide valuable meta information to search engines trying to connect users with your content. Doing this consistently can push your site higher in Google's search results.

The recent push towards mobile web sites ties in well with the accessibility concept also.  What might work fine on a 21" monitor with a good mouse does not usually work on a 4 inch smart phone screen. Web accessibility recommends using large "click targets" to support users with impaired motor skills. But keep in mind that large targets will help all your users when they are tying to get something done on your web site using a phone or tablet.  Especially if they have to poke at the tiny screen with their finger on a bouncing commuter train.

If you are interested in learning more about Web Accessibility guidelines at the W3C's Web Accessibility Initiative site.

Protect Yourself Against Surprise Cloud Hosting Fees

There are lots of great things about hosting a web site or service in the cloud, but price stability is not one of them.  There are numerous ways a small project can get hit with a bill that is many times more than you would expect for the month.  Here are just a few scenarios:

• Your web site goes viral and suddenly you are getting thousands of downloads of your hosted content. Not a big deal if you have a static website, but if you have some large files that are being downloaded, you can burn through bandwidth rather quickly.  A 50MB file is only $.006 per download, but 50,000 downloads will cost you $300.
• Some of your custom code goes wild and starts making excessive data transfers in/out of your availability zone.  Or to a lesser extent, just starts burning through I/O operations.
• You accidentally start an AMI with extra charges above the usual per hour rate.  For example, right now a Large Windows instance is $0.46 an hour, but a Large Windows Instance with SQL Server is $1.06 per hour.  That's an extra $432 a month you might not notice until you get your statement.  Other AMIs could potentially have a larger markup too.
• Your site is compromised and very bad people start using it as an open FTP site.  Amazon has a huge pipe to the public Internet and you could end up paying download fees for 10,000 copies of a HD movie hosted on your server.  This would cost you thousands of dollars and it is the one I'm most worried about personally!

The thing that seems strange to me is that Amazon has yet to put in a publicly available kill switch or even an alert system for crazy cost spikes.  People have been asking for on the forums frequently since 2009.  On the plus side, I have found some info in a post on the BitNami Blog that Premium Support customers can use the Cloud Watch service to monitor charges, but it doesn't look as simple as it should be, and it's not available to the 'little guys.'  Regular users can setup free Cloud Watch alarms based on certain metrics like Bandwidth or EBS reads, but not on the actual charges you are incurring. 

So, what to do?  You could give up on the cloud for small hosting projects and stick with a fixed cost VPS provider. Or, if your site works with the right technology you could use a cloud provider like Heroku that gives you a fixed cost and a soft bandwidth limit of 2TB.  Or for people who trust in their OCD, you could manually check your billing summary page every day. None of these are great solutions if you want want Amazon is offering.

There is one web service I've used personally called Cost Alarm.  It does exactly what I want, monitors your overall AWS bill and sends an email messages when your charges reach whatever level you specify.  I use my mobile carrier's email to SMS gateway so I will get a text message should there be a problem. 

Cost Alarm is still in beta test so it is free for now.  When the service does go live, the proposed cost is about $5 a month for a each AWS account monitored, which I think is reasonable.  I have ran some test cases and found that it does exactly what is advertised.  I wish that Amazon never created  a situation where a site like Cost Alarm was necessary.  It is really a feature that should have been baked into AWS from the start.  Given the situation though, I hope that Cost Alarm does well and keeps offering this necessary service.

 

Debuging Dynamic JavaScript

I have always been a fan of integrated debuggers.  They really help me think about tricky bits of code and verify that my solution is correct.  While testing my code I like to use a debugger to watch all my variables and boolean conditions.  Sometimes during a walk through like this I will see some problem with a variable value that isn't manifesting to the user with any of my chosen test cases but could occur in the real world.

With JavaScript becoming a larger and larger part of my code base it's not as easy to do this.  There are some great tools that help with debugging JavaScript, and my go-to tool is FireBug.  It seems to do nearly everything for client side debugging: DOM inspection, source viewing, line by line JavaScript debugging, etc.   90% of the time it does what I need. 

There is one place that I have had trouble though:  Finding what bit of random JavaScript has been dynamically bound to the DOM.  Often the code that binds an anonymous function to some element wasn't even loaded with the page but was pulled in later due to a dynamic condition.  I am very happy to have found a tool to help with this: Visual Event.  It works by highlighting on the screen all the elements that have JavaScript bindings. When you hover over the element you can even see the code that is bound to it.

Here is what it looks like when I run it against the front page of NITRC:

If you run into some code using a lot of client side JavaScript magic, give it a try and see what you think.

Are you having problems with VirtualBox giving out of memory errors?

In a previous post I described how I use Virtual Machines while doing software development.   A few weeks ago I started getting memory errors when I launched a VM without closing down any other VM I had running.  Specifically, I would get a VERR_EM_NO_MEMORY error with a result code of 0x80004005.  I had never seen anything like this before and assumed that after I rebooted my computer I would see it again.  Unfortunately, it did not go away.

Running a single VM at a time isn't much of a hardship, so I didn't jump on this problem right away.  Eventually I did some research though, and it took some digging to find any relevant information. Since I had trouble finding information on the problem, I thought I would document what I found in a way that search engines could easily index: a blog post.  That way anyone else who is having a similar problem is more likely to find the solution.

Here is a link to a VirtualBox bug report on the issue.  The short story is that this problem can be resolved by killing the Google Crash Handler process.  Once I did that, I was able to start up numerous VMs as usual. 

It turns out that, for me at least, the Google Crash Manager is being launched by Chrome.  I found a related Chrome issue report.  This page explains that you can prevent the Crash Manager from running when you start Chrome in the first place.  Here are the steps:

• Hit the wrench button to the right of the address bar to pull up the Chrome menu.
• Choose Options from the menu.
• Choose "Under the Hood" from the left hand menu.
• Find the check box in the Privacy section that says "Automatically send usage statistics and crash reports to Google" and un-check that.

I expected the problem to be resolved before I had a chance to blog about it, but so far I am still seeing it.

Virtual Development Paradigm

I use virtual machines (VM) for nearly all my development.  When I am doing any serious work, the first step is to fire up VirtualBox on my desktop and launch the VM that I am using for that project.  If you don't do this yourself, it may seem kind of silly to run a desktop OS inside your desktop OS so you can run a few standard development programs.  Once you start doing though, you quickly realize that there are a lot of benefits.

• Environment Stability - No matter what crazy software you install on your desktop computer, your real work environment in the VM can stay clean and focused. Tool upgrades needed for one project won't break things in an unrelated project.
• Disaster Resistance - Eventually in every computer's life there comes a time for major maintenance.  When you decide it's time to start from scratch with a clean OS, having your development environment in a VM means that you don't have to reinstall and configure all your critical software to become productive again.
• OS Independence - It's not cool to like Windows, but it is what I'm used to.  Even if I liked a more hacker approved OS like Linux or the aesthetically pleasing OSX, it's not going to match every one of my production environments.  I write software that is deployed to different platforms and it's great when I can match my dev environment to the target, be it Windows or some particular version of Linux.
• Maintenance Mode - A lot of my projects are in a fairly entrenched maintenance mode.  So once or twice a week I have to jump in make some small change, test, deploy and then get back to other things.  Having a VM devoted to that project lets me leave my IDE, command shells, FTP clients, etc. open to the project.  When I'm done working I suspend the VM with everything open and ready to go the next time I have to make a tweak here or there.
• Tool Partitioning - As I've said, I work on a lot of projects a once.  Some for work and some for fun.  Some use PostgreSQL, some MySQL, some MS SQL Server.  Installing every tool, every front end, every database on a single desktop PC is not at all a good idea.  By using one VM for each paradigm I don't have any single system weighed down by every flavor of technology stack. 

There are a few free tools that you can use to get started with VM based development.  I use VirtualBox but VMWare makes a number of free products that fit the bill.  If you use Linux on your desktop you have a number of additional options like Qemu and Xen. 

Licencing can get a bit tricky.  Obviously for Linux based VMs you don't have to worry much.  For Windows the worst case scenario to keep in strict compliance would be to purchase a licence for every VM.  A cheaper option for an independent developer is to join Microsoft's TechNet or for business users to look into something like MSDN. The licencing for OSX on virtual machines is tough. The shrink wrap licencing agreement for the desktop versions of OSX say that you can't run the OS on non-Apple hardware.  From what I understand, OSX VMs even on Apple hardware are a no-go too.   On the other hand, you can run OSX Server in a VM on Apple hardware.  I am just starting to do a little development for iOS and to get your apps into the Apple store you need physical Apple hardware.  My solution so far is to use a Mac Mini for dedicated mobile development.  Not ideal, but it does at least work for me.

So that's how I handle my development hardware.  On the server side TCG uses a number of VMs for various tasks.  Some are kept on an internal server farm and some are hosted with other providers depending on the particular needs of that system.  I'm sure most developers have at least dabbled in virtualization.  If you haven't done it, give it a try.

TCG Customer Named Federal 100 Award Winner

Christopher Friedline, program manager for the Department of the Treasury/Bureau of the Public Debt Budget Formulation and Exectuion Manager (BFEM) program, has been named a 2012 Fed 100 awards winner. The awards, which are sponsored by Federal Computer Week, recognize 100 individuals in government and industry who have played pivotal roles in the federal IT community. TCG supports the BFEM program with a range of services, including program management, business analysis, program implementation, systems architecture and development, and operations and maintenance. To read the full press release click here.