An Analyst's Hunt for Red October
OR How I learned to stop worrying and love robots that look like people.
I measure how people flow through websites.
I figure that if you watch how people walk around the store, you can infer what they want, and change the layout of your store to make things easier to find. I’m always on the lookout for problem spots, where people suddenly do something totally unexpected, which means that the word we keep using doesn’t mean what we think it means.
Wallace Shawn in The Princess Bride
I like to think I’m pretty good at it—but three times in the last year I’ve been hugely embarrassed. In all three cases I presented user behavior analysis to clients, only to have to retract later.
The culprit: robots that look like people.
All three times, the companies I engaged with had implemented some kind of automated monitoring system to make sure their site was working at all times. One was through a vendor, but the other two were home-grown solutions a developer had kind of thrown together on a whim.
One of these scripts was set up to start on the Home page, advance four pages and launch our flash app, then quit. Another one entered the site, picked a product, and advanced through the checkout process to the very last page, and then quit.
Now, companies that offer these services will tell you that they’re operating a such a low level that the trail they leave won’t affect your overall results. But they forget that real people naturally fallout as they advance through your site funnels, but robots push forward with their indomitable spirits (read: they have no spirits).
Imagine you have a 2% conversion rate, and the shortest route from arrival to purchase is 7 steps. Statistically speaking, it’s the equivalent of losing about half of your visitors on each step.
Now think of a robot who looks like a person who’s monitoring your checkout process. His creators told you he’d fly under the radar, and would represent less than 1% of your total site traffic. But since that 1% doesn’t fall out, he will be vastly more detectable the deeper into the site he goes. If he’s programmed to run six of our seven steps, 1 percent on entry becomes 2 percent on the next page, until it’s 32 percent on page 6.
Robots skew the end of the funnel more than the start.
That may sound like a dramatic exagerration, but I promise it’s not. In one of my three cases the robots accounted for 80% of all traffic on the last page of the checkout process. You can imagine what it did to my analysis. I hastily asserted that something had to be technically wrong with the last page of our checkout process, because you just can’t dissuade 95% of the people who get to the last page to quit, even if you try!
Safeguards
Here are some ways I’ve used to detect these Frankensteins (but please let me know if you can think of other good ones).
- Domains/ISP Reports. Traffic from site monitoring companies like Keynote, Gomez, and ObservePoint are easy giveaways. Set up some alerts today, or set up filters to remove that data before it starts pollutingyour reports. Another ISP you should keep a close eye on is that of your own company—although it doesn’t always mean automated traffic. Trend these over time to see if anything jumps out.
- Full Path Reports. If you see that a multi-page path is among your top ten, that should raise your hackles. Trails left by humans always start with single-page visits, and then grow from there. An elaborate, seven-page path breaking into my top ten is what tipped me off to one of these situatoins.
Solutions
If your site monitoring efforts are coordinated with your site analytics, you can pre-configure these automated visits to suppress the analytics call in the first place. But I also noticed that Omniture offers a VISTA rule that will exclude known monitoring programs for you.
Google Analytics has a new Intelligence tool which silently monitors reports you may not even be looking at, and tells you when anomalies occur. But what I’m really looking for is something I don’t think anybody’s created yet: a tool that listens carefully to the normal buzzing around your website, and notices when things seem too normal.
Do you remember the movie Hunt for Red October? A submarine technician had listened so carefully to his sonar that he could detect patterns that were created to sound like the sea, but were really generated from an enemy sub. I don’t think I’ll ever be that good, so I’m hoping that somebody will create a machine that will do it for me!
Sean Connery and Courtney B. Vance in Hunt for Red October
