Now, the Bad News

News hit the Internet pretty fast today as one of our QA offices suffered quite a blow. As part of our company's plan to return to profitability, the vast majority of our testers in Champagne,IL will soon be out of work. Being on the Nintendo Compliance team, I had no direct contact with them as far as projects are concerned (they concentrated on PS3/360/PC testing), but it's never easy to hear that co-workers have lost their jobs.

Best of luck, guys, and we all hope that you land on your feet soon.

Fun Friday: QA Funnies Edition

Game Developer's Matthew "Wasteland" has taken the task of infusing some funny into their industry leading magazine. His back page article this month tackles QA bugs, but not the good kind. Here's my favorite:

1. Put the game in the disc tray and start playing it.
2. Walk 56 steps to the left and drink the green potion.
3. Explore the caves and proceed to the point where the wizard talks to you, then skip the dialog after about the third panel. I forgot what it said.
4. Hit the attack button repeatedly for five minutes.
5. Pause the game and leave it on the Pause menu for about as long as it takes to make and eat a microwave burrito, I think like 15 minutes.
6. Unpause the game, turn on all of the cheats, warp to the vampire's lair and shoot the panzerfaust at him over and over as fast as possible.
7. Notice that the game has frozen.

I'm sure the developers response to this bug was either, "We'll fix it in the sequel," "But LotCheck/Certification didn't find it," or " Do you think we can get this waived?"

Summer Has Finally Ended

Posting has been light as of late, but I'm glad to say that while Summer is officially over, so is our ridiculous summer in QA. Since, as a company, we try to submit about 90% of our titles between the months of June and September, needless to say, saying it was crazy is an understatement. But it's appears to be finally over. I helped get SmackDown Wii out the door in early August and it passed Nintendo of America first try with Wi-Fi (I don't think too many other publishers can say that). The two other projects I was working on our now out the door as well as of today. August saw us work some crazy overtime but we can see the light at the end of the tunnel and Fall should be significantly less stressful.

On the other hand summer was as usual, light on releases, so I was thankful for that. I finally crossed the 15k GamerScore threshold thanks mostly to Civilization: Revolution and Lego Indiana Jones. Civ:Rev was a lot of fun and hope to back to again soon. This is saying a lot because I absolutely loathe real-time strategy games. This game was fun and accessible and did away with a lot of micro-managing. The game was deep enough for me to feel that my decisions, for better or worse, were actually affecting my society without burying me in data and options. Lego Indiana Jones was a blast to play even though I was uninterested in the Star Wars games after playing the demo. A lot of the achievements were a grind but most were cleverly named and the humor throughout the game kept me motivated.

Xbox Live Arcade also saw a lot of releases in August. I started playing Braid and absolutely loved it until a save killing bug popped up and wiped my data. Bionic Commando was awesomely retro and brutally difficult. Castle Crashers is beautiful and humorous, but the same bug popped from Braid and I'm waiting for the patch. Needless to say, I will be returning to all of these games soon.

I just picked up Viva Pinata: Trouble in Paradise, mostly because the original was a guilty pleasure and they priced it at only $40, I could not pass this one up. The game was everything I hoped for: more adorable pinatas, more environments, more everything but the same addicting game play with a few new twists. However, a word of warning: watch out for Hemlock! After cultivating a healthy garden of pinatas (some of whom I had become quite attached to), I became afflicted with a massive outbreak of Hemlock. This weed spreads incredibly fast, attracts pinatas like nerds to boobs and makes them poop sour candy when they eat it. After about ten minutes my entire garden population was decimated; all of my pinatas became sick faster than I could help and when I ran out of money, Dastardos swooped in and killed them all. This was one of the few times I absolutely despised autosave.

In the end, I skipped Too Human and Force Unleashed. I'll probably borrow them from friends, but with the likes of Fable II and Fallout 3 coming out, who knows when I'll get to them.

Oh, and go buy de Blob if you own a Wii. It's awesome and I want to work on more games like this

Holiday Season Bloodbath

While it's good for my wallet, I'm sad to see that Afro Samurai has been pushed back till January 27th, 2009. While an official release date may not have been announced, many retailers had it pegged for an October or Fall release, a long with just about every other title we know about.

I will never understand our industry's continued dependence on the holiday quarter for release schedules. Firstly, with the escalating costs of development and vast number of choices all competing for limited funds out there, makes little business sense to me to force a game on the public. Then, year after year, quality but lower profile games get over looked by the masses further enforcing the downward spiral of publisher reluctance to produce original IP.

I also see a drastic effect on overall quality as well. We have just over 70 skus in test right now. And that is just Nintendo platforms. With a nine man team, every game is not going to get the test time that they need. Granted we aren't testing the functional stuff, but those teams are stretched thin as well. The rush to hit dates is on and quality is the first to be sacrificed. Instead of pacing ourselves for the year and bringing more quality to store shelves, we burn ourselves out in the summer to get anything we can to Fall shoppers.

But, obviously, we aren't the only publisher with this mentality and I wish a few would start seeing that there are other months throughout the year. Especially when we almost always go from January to August with high profile releases very few and far between.

October was already going to be expensive with Fable 2 and Fallout 3 coming out. There's also Guitar Hero World Tour and Dead Space. Afro Samurai looks like a very fun game but it would have probably been relegated to a game I borrow from someone instead of full purchase. Being released in January may be the best decision their publisher has made.

Layoffs Suck

This is too bad. It appears that Sandblast Games (formerly Cranky Pants) will be closing after massive layoffs. Evil Dead: Regeneration was my second project as a tester. The game was fun if a bit repetitive. I haven’t seen much from them since, which might explain their situation, but it still sucks.

I was talking to one of my coworkers (a test lead) about this and he commented on how it seems like there are always news stories about massive layoffs at developers but that he feels safe in QA because that thing doesn’t seem to happen to us. Maybe it does happen and we don’t hear about it as much, but since I’ve moved beyond the level of a temp three years ago, I have yet to worry about my job security. I guess I should consider myself lucky.

Insert Foot in Mouth Week

There seems to be a running theme for developer quotes this week. Firstly, Greg Donovan, Producers for Saint's Row 2, lowered every ones expectations by reminding us that the game "won't be bug free" since no game is. Working in QA, I know the truth far too well on the bugs that get through to the final product but stating this up front is not getting off on the right foot.

Not to worry though, because Spark CEO Craig Allen one upped Donovan by telling us why most of us don't like his game, Turing Point: Fall of Liberty.

Turning Point, the idea there was really a high concept idea. I think that we're finding that the people who look at it like, "Is this going to be fun, something I can drop into, in a world I haven't seen before and have a good time?" really love the game. People that are looking at it with more of a critical eye of "Is this game revolutionary, is it doing play mechanics I've never seen before, is it driving new technology?" it was never meant to do those things. If people look at it in the right way, I think they really enjoy what they see. You can certainly look at it and say it's not revolutionary, so those opinions are certainly a lot more critical.

I guess that explains why I didn't like the demo at all. Not only am I not smart enough to get the concept but this first person shooter is not for core gamers. Duh!!!

Former Co-worker on Xplay

Seen in the video above is Ali Zandi, a former co-worker of mine who has moved on to bigger and better things over at Pandemic. We tested Smackdown '06 together and it looks like he doing good for himself. Keep up the good work Ali!

Assassin’s Creed Bug Helps Earn Achievement

I have finally gotten around to playing through Assassin’s Creed and I love every moment of it. While many of the mission types are repeated, they never feel stale or repetitive. The AI may follow fairly static paths but the crowd seems so dynamic that a mission is never the same when retrying it. Additionally, assassinating someone with the hidden blade never gets tiring or old, it is always satisfying!

One of the achievements requires the player to remain in a fight for ten minutes. While that doesn’t sound like a long time, that’s a lot of blocking or running in a circle. The game has a fairly well known reputation for being a bit buggy, nothing game breaking as far as I know, but it’s a big game and they’re bound to make it in to the final code. At one point while riding my horse through the kingdom, I was attacked by one of the Templar Knights. This attracted several other enemies and I decided to high tail it out of there. I had my sword out and was running at full speed straight at a wagon full of hay to hide in. I attempted to jump into the wagon at the last second but instead of jumping off of the horse, Altair stood up and froze. I lost most functionality and I couldn’t get off of the horse, but I did become invincible. Since the enemies were still trying to attack me, I thought I would put the above achievement to the test. I got up and went and made lunch. When I came back, the achievement had been unlocked! Sweet!

Apparently I’m not the only one who has come across this issue as I found a YouTube video confirming that it’s probably not a fluke but a fairly repeatable issue. I assume that it was not severe enough to fix, probably causing more issues than was worth. In the end, I had to exit out of the Animus and reload my memory but fortunately I had just saved and didn’t lose a lot of progress.

On a side note, the game has many collecting achievements with the flags and Templar Knights. I traditionally hate these achievements but I came across an excellent site with very good maps for all of their locations. With this help, I’m well on my way to a perfect 1000/1000 in this game.

Fun Friday: The No Video Edition

This week's Fun Friday does not include a video. To change things up, I'm quoting an article that I found on Digg earlier this week that actually had me rolling. In an article that cited the top ten hot computer driven careers, they included video game developer. So far so good, I wholly agree with that. Even though times are tough, the industry as a whole is having tremendous success. However, there was a link to becoming a tester that was full of wisdom such as,

Is your Play Station controller soldered to your hand? Did you buy a Wii before anyone else? Do you like nothing better than spending hours in front of the TV playing Halo? Instead of playing those games for fun, why not play them for a paycheck? Get paid to pay video games!

Thanks for perpetuating the stereotype! One of the hardest things for gamers who become tester to learn is to stop "playing" the game and actually test it. But the article is not done yet. Since the job is mostly fun and games, how much could this unbelievable job pay?

Because video game testers are considered entry-level positions in the gaming industry, they are often offered jobs on an as-needed basis rather than a regular nine-to-five position. But this doesn’t mean the job doesn’t pay well, because it can—anything from $10 to over $100 an hour. Pretty good for a video game job!

The emphasis is mine, but wow. I must be working at the wrong company if other testers are making nearly ten times what I make.

Puzzle Quest Postmortem: Required Reading

Puzzle Quest developer, Infinite Interactive, posted a postmortem for their surprise hit in the September issue of Game Developer magazine. Of all the postmortems that I have read in this magazine, I was most impressed, and inspired, with the philosophy and wisdom the developers shared in the article. While many postmortems seem to spout the same "what went wrong" and "what went right" issues while creating a game, Infinite Interactive touched on many things that I have seen, and have been bothered by, during my short time in the industry. I have yet to play the game, but now that it is out on Xbox Live Arcade, that will be remedied soon.

This is the first postmortem that I can remember that stated that taking their time to finish the game was one of the things that went right. The game may have only been released on the DS and PSP, but polish is what seems to separate the quality titles of the current generation from those that are hoping to make a quick buck. I know that most publishers have quarterly goals to meet and that developers are usually working on tight deadlines and even tighter budgets. The problem is that it shows in the number of quality titles that are released throughout the year. Infinite Interactive spent nearly two and a half years making the game the best it could possibly be. They were rewarded by not being able to keep the game stocked on shelves.

Of the things that went wrong, "underestimation" is something that is not exclusive to Infinite. Every new developer we work with, and even ones that we've worked with a lot, seem to make the same mistakes. Something that I have suggested and pushed at work is echoed in the sentiment of Puzzle Quest's lead designer, Steve Fawkner.

My advice to any teams just getting into Xbox Live Arcade development is to read the TCRs over and over until you have them committed to memory. You may even want to run your design documents (especially the UI designs) past an Xbox Live Arcade Q/A team before implementing anything.

He talks about Live Arcade but I think that would be useful for anyone developing on the DS all the way up to PS3. It never fails that a game will come into test with only about two months till submission, but every Guideline fails. It then becomes a test of wills on whether or not to fix the issues or wait to see if they're found in submission. I can only imagine the time that would be saved if games were built around the guidelines for a particular system as opposed to throwing it together on one system and then attempting to shoehorn it onto another.

Lastly, he gives props to his "two excellent test teams." Being the lowest rung on the pole is not easy. At the best of times we can feel under appreciated and at the worst, seemingly blamed for everything. There is no glory to be had in Q/A and even the best teams are fallible as Fawkner points out that a heinous, if not serious, bug made it into the release version. It makes me happy to see developers chalk it up to an enormously complex game and not lazy testing.

Developers like Infinite Interactive seem to be few and far between. They have tremendous passion and the will to succeed, and it shows in their success.

Testing in the Open World: Crackdown

"Everyone knows that testing a game thoroughly is essential to its quality," is the debatable first sentence in a Gamasutra post from Microsoft's Gamefest concerning a forum on how they overcame testing the massive size of Crackdown. While the article is sparse in detail, it provides an overview of what steps were taken to ensure that the game was thoroughly hammered. Essentially, due to the overwhelming scope of the game world that was compounded by the fact that much of the game rose in the y-axis as well, much of the testing was automated. Environmental bugs, including the placement of items and graphical seam issues, numbered well above the 10,000 mark. I have not seen many multi-platform games get that high in total, let alone one bug type.

Weren't We On The Same Team?

I don't personally have a subscription to Game Informer, but there are so many people at work that do, I usually get a chance to at least flip through the pages. The current issue, August 2007, includes a quick article titled, 'The Ties That Divide.' The article sheds some light on the inner workings of the publisher/developer relationship. It is no secret that these alliances can be quite uncomfortable under even the best of situations. Still, it never ceases to amaze me how easy it is for each side to quickly turn on each other. Unfortunately, as the article points out, it largely comes down to money. In a tug of war match that becomes the milestone system, it is largely forgotten that we are all on the same team, with, assumably, the same goal: to make a great game and a lot of money.

The article quickly mentions that one of the first things to go when the ax begins to fall is QA. As one anonymous producer points out and the author elaborates,

'The money people are like [Quality Assurance] doesn't make us any money. Shipping a game and selling it makes us money. [Developers] being happy after the fact don't make us money.' Many developers told us that this stage is crucial because it's when a game comes together and gets polish - which often separates the good titles from the bad.

The majority of the time I feel like we are a third tier to contend against the developers and publishers, despite the fact that we technically part of the publishers. We are still seen as the whipping boy; but lack power to fight back against the publisher's money or developer's superior knowledge of the game. It's nice to read that there are developers out there that appreciate what QA can do for a game, it is just not what I see on a day to day basis.

Far too often, we deal with developers that don't have the time or money to fix issues, were so cheap to begin with that they don't know how to fix an issue, or they want the game to work their way and disregard what Nintendo requires of a game. On the other hand, our company, as the publisher, is pushing dates down our throats and berating us for finding issues that threaten those dates. The article suggests that Marketing can be one of a publisher's greatest strength that they bring to the table. They're fine when they're pushing your game to magazines and websites, making sure it's on store shelves, and I like the idea of them being "slightly behind the developer itself in knowing where the game is." The quickly become a nuisance when they want to affect code, features or want European versions (they play games in Greece?) to ship at the same time as North American versions. It's quite disheartening, to be honest.

I would just once, like a Publisher to afford the developer the time and money to take some risks and add a bit a polish, even to our licensed games, to improve the product in the long run and to make a better game for the player. I would like for the developer to build there game around Guidelines instead of trying to shoehorn them in at the last minute and not argue with us about functionality that we have seen a hundred times and is not present in their game. In return, we, QA, will communicate our issues early, efficiently and with literacy. The perfect three tiered team, working in harmony.

Oh, who am I kidding? That would probably cost too much money.

Bottom of the Barrel

As clichés go, QA as a dream job is one of the most repeated in our industry. Joystiq and Kotaku both posted an article from the Seattle Weekly where one of the writers took a job at Volt, a local company contracted to test hardware for Microsoft and the 360. It's an interesting read, if not wildly extreme. While it does describe the conditions of a Quality Assurance department, this company looks like a Nazi concentration camp compared to my company. But the author brings up some interesting points and one quote in particular struck a chord with me:

"I think the [video game] industry in general, they want addicts," says Dr. Hilarie Cash, founder of Redmond's Internet/Computer Addiction Services. "It is to their economic benefit to have people really hooked on their games. That there happens to be a pool of addicts out there who will work for peanuts, like methadone treatment, is to their advantage," she says.

Quality Assurance always appears to be an after thought, an unfortunate cost that must be begrudgingly provided before a game can ship. It is a vicious cycle between the other departments loathing for us and their inability to provide an environment for us to succeed and prove our worth. We are constantly belittled for holding up projects by doing our job and then accept the full brunt of the blame when things don't go well. And yet, there is no active catalyst for change, there is no open discussion for how to improve what we do. So we stick with the status quo.

Reviling our ability to perform and then posting a job that only pays just above the state minimum wage only encourages more of the same. We will continue to draw from the same pool of mindless addicts instead of analytically thinking, intelligent people who could probably be enticed by a moderate increase in the starting wage. At the very least, we could be more discerning about who we choose to hire and hopefully weed out the bad apples that give us a bad name.

Unfortunately, we practically set up those that do make it through the loose screening process for failure with a training system that is antiquated and ill focused. Which brings me back to the quote; we don't want mindless addicts, we don't want drones. This defeats the purpose of QA. If one fails to see the details and becomes comfortable with the mundane, things begin to seep through the cracks. We need to stop pulling from the bottom of the barrel.

Quality Assurance can arguably be one of the most important aspects of game development. After all, there is no such thing as a B-Game.

Skills to pay the Bills

I happened to notice that Neversoft is hiring Guitar Hero Testers for the next iteration of the game. The description list qualifications such as being ready "to rock!" and passing a Guitar Hero "proficiency test." If hired, do you get a dressing room and the ability to make outrageous demands for what is put in there?

In all seriousness though, more power to anyone who is qualified. I'm almost complete with five stars on medium and fear moving on to Hard, let alone Expert. This is a skill and a great opportunity to get in the door. I believe Neversoft has a record of hiring people as testers only to promote them due to their testing/playing skills on the Tony Hawk series. The next thing this person knows, he or she could be selecting the songs for the set list in Guitar Hero III. And that would definitely pay the bills.

Graphs are just as good as pictures

In the most recent issue of game developer, Midway's QA Manager, Paul Sterngold, describes a situation in which a miscommunication between his teams and their outsourced QA partners led to an unexpectedly large bill at the end of the test cycle. He goes on to describe Midway’s solution, with the support of Amritt Ventures, which involves a series of graphs that quickly describe a project's status and performance statistics of the QA test teams.

Being in QA, I spent some time reading over the article and attempted to analyze the graphs, hoping to garner some information that may prove useful to my own company. After a while I noticed a trend that reinforces a one of my fundamental beliefs about this industry: more hours do not produce results. A quick comparison of the upper-right blue graph to the two yellow graphs indicates that as tester hours go up, the number and quality of the bugs go down. All the while, costs appear to be going up as indicated by the green graphs. Granted, I am deciphering these graphs with only the data and information I can glean from them. There may be more to the story, but as my title suggests, graphs are just as good as pictures.

Quality of life is very important to me, as it should be to all in the business. I have not been in the industry too long and I am fortunate to have worked myself into a position that does not require extensive overtime. I have done my share of late nights, but it has been my work ethic that has earned the respect of peers and advancement to my current position. I, like the people I respect in the industry, am willing to work as many hours as it takes to get the job done. If I feel I have given my all in a normal workday (by which I mean eight to nine hours), I see no need to stay any longer than necessary. And still, despite overwhelming evidence that supports it, people continue to believe that more is better when it comes to working a QA team. I, on the other hand, feel that a team of well-trained, motivated individuals will consistently out perform low-paid, under-trained, and over-worked testers any day of the week. Except on Saturday and Sunday, their job will be done within normal working hours.