One of my rules when writing a product review is to never ask the price until the end. I’ll do everything I can to stay away from the figure after the “$” because I don’t want it to affect my perception of quality. Not so much in studio products, but in the audiophile realm, the idea “it’s expensive so it must be good” is king and drives sales. Now this isn’t to say that something expensive isn’t the absolute top of the heap, but it’s not always true.
Case in point, I went to the Consumer Electronics Show in Las Vegas this year for the first time and it was an eye- and ear-opening experience. My engineering buddy Dave Rideau and myself jumped from room to room listening to incredible systems with audio playback from vinyl, analog tape and digital sources through the priciest players, converters, amps, cables and speakers. With only a few exceptions, the brands were nothing you’d ever see in a Best Buy or Guitar Center, and many systems cost as much as a small house.
There were rooms we visited where the playback was making us smile and others that were categorically unimpressive, where the top end was strident, bottom boomy or there was that intangible feeling that something just wasn’t right. But the one thing they all had in common, at least if you read the product material, was they made the best stuff on the planet.
So where is the truth about quality? How do you separate what you see, hear, know (or knew) from the hype? And even more important, how do you keep your own internal prejudices—like “it’s too expensive,” “too inexpensive” or “I don’t like that brand”—from souring your evaluation?
To get some perspective, I talked to JBL’s Peter Chaikin about the methods the company uses to create speakers. It turns out that JBL has a unique and very scientific way of reducing sensory input when testing speakers for consumer “lifestyle” products, automotive and, of course, professional live and studio audio.
It all started in 1991 when Dr. Floyd Toole came to Harman as VP of acoustics. His mandate was to bring a scientific method to the development of Harman products. He believed you could measure quality and design to those qualities, getting close long before the consumer ever heard the speaker.
Harman built a Multichannel Listening Lab (MLL) at its Northridge, Calif., campus and other locations around the world to collect data on listener preferences. The methodology has been to bring people through the facility and test their responses to various playback systems, including JBL’s competition. The company buys off the shelf and puts competitors into the mix. Those invited to listen include marketing managers within the company, college students, product reviewers, professional engineers and more. Through the process, JBL attempts to prove or disprove what’s true about listener preferences.
How you test is the key to getting good data, and Harman’s test is double-blind. In the MLL, Harman created a device called the speaker shuffler, a hydraulic machine that operates behind a visually opaque and sonically transparent screen, and lets you swap up to four pairs of speakers up to 200 pounds each and put them in the exact same position, all within three seconds. It removes what Chaikin calls the “nuisance variables” that can color perception, such as logo, brand, look of the product, color, price and, most importantly, position. Any speaker will sound different if it’s placed even a foot to the left or right, affecting the listener’s judgment.
They use broadband music for playback or invitees will bring their own music. In four separate trials, the listener rates each speaker from 1 to 5 and enters comments. The speakers are switchable from a touchpad, on which they are given an A, B, C or D designation. The listener doesn’t know which speaker is which, and the music played is random. Also, the letter designation for each speaker changes from trial to trial. So the speaker on button A playing music track 3 in test 1 may be the same, or different, in tests 2, 3 or 4.
Harman then comes out with a rating for each speaker, as well as a rating for each listener and how reliable their responses are. Here is what’s remarkable. From all this testing, it turns out that all listeners from Pros to Joes, in all countries, tend to prefer a speaker that measures flat. Not what I, or Chaikin (who comes from a pro engineering background), would have thought. So using these results garnered from frankly the most fair-testing process I’ve ever heard of, Harman creates speakers that will please the widest variety of listeners.
This is not to say that JBL is the only company with bead on excellence. I’ve visited both the Genelec and Focal manufacturing facilities and found them to be exceedingly passionate about what they do. All speaker manufacturers test their products scientifically and perform extensive tests in a variety of environments, anechoic and not. But the MLL’s ability to objectively assess a listener’s subjective evaluation is unique and one we can learn a lesson from.
The truth for me is that your ears are the most reliable resource available when making product-buying decisions, as long as you deprive yourself of hype and try to reduce your own internal bias based on past experience. Without your own personal MLL, it’s a tough thing to do, but knowing what keeps you from making better decisions is a great way to up your buying skills.
