Let 1,000 Frequency Responses Bloom

    Audio: Listen to this article.

In the era of cable news and social media it might be hard to believe, but journalism has a code of ethics. Among other responsibilities, journalists are expected to “label advocacy and commentary” and “show compassion for those who may be affected by news coverage.”

These obligations might seem obtuse when it comes to product reviews. Reviews have real-world consequences yet are inherently opinion-based. A rave might persuade consumers to part with their hard-earned cash. A pan might imperil the livelihood of the producer. Given that, experts have puzzled over the ethics of reviewing everything from art to food. Each genre has its own quirks. But in every case, an ethical reviewer must be fair to consumers and producers, alike. Making clear that the review represents opinion, not fact, is central to that fairness.

I think a lot about these obligations when I review DACs, amps, headphones, and IEMs here on Audiophile Style. I’d rather write with an even-handed matter-of-factness than throw bombs. I try to take practicable steps to reduce my biases. I level-match. I listen multiple times before I measure. I also confront myself with opposing views after I have formed my initial opinions. If I think I’ve missed something, I go back and relisten or remeasure. I also recommend that readers consult other measurements, not just mine. If I’m saying that I don’t like something that many other reviewers have praised, I don’t want to hide that fact. Ultimately, all my reviews can convey is what I prefer and why, not what’s better or worse in an absolute sense.

For the past several months, I’ve been working on a headphone review. I already have my own listening notes and measurements, and I’ve started to peek at what other reviewers are saying. Naturally, I was interested when a reviewer with a GRAS 45CA measurement rig, which is much more advanced than my meager miniDSP EARS setup, reviewed the same product. But the measurements weren’t what grabbed my attention when I started reading. It was the strident tone of the reviewer. He declared that his tastes are correct, the product’s tuning is wrong, and those who disagree are fools.

The reviewer prefers what’s commonly known as the “Harman Curve,” a suggested frequency response for headphones and IEMs. There are many excellent summaries of the Harman Curve out there. In short, the Harman company’s research started from the logical premise that a good headphone or IEM should “simulate the in-room response of a well-designed loudspeaker system calibrated in a reference listening room,” as Harman’s Sean Olive put it. Thanks to the research of folks like Olive and Floyd Toole, among others, we know more about what makes for a good-sounding speaker than ever before. But making a pair of headphones or IEMs sound like that is easier said than done. There are quite a few ways to measure speakers’ in-room response, and a combination of factors — including anatomical variation and room reflections — influence how the listener experiences that sound.

To develop Harman’s headphone and IEM curves, Olive and other researchers began by measuring the in-room response of good speakers using a dummy fitted with a GRAS fixture. Using that response as a baseline, they conducted a series of experiments whereby listeners could adjust the bass and (sometimes) treble level of headphones and IEMs to their tastes. The Harman Curve is, more or less, an average of what those listeners liked. Specifically, it represents the frequency response that 64 percent of participants preferred. The remaining 36 percent were divided between those who liked more bass (15 percent) and those who liked less (21 percent). The Harman headphone curve has been revised several times, with the general trend of increasing bass below 100 Hz. The in-ear-monitor version of the curve, in turn, has even more low end than the revised headphone curve.

The Harman Curve has not been met with universal acceptance. It’s been criticized for a variety of reasons, both methodological and ideological. Some critics think the simple bass and treble controls given to the Harman listeners weren’t fine-grained enough. Others object to the fact that the largest Harman study EQ’d a single pair of headphones to match the frequency responses of other pairs instead of using the actual headphones. Still others just don’t like how it sounds.

I think there’s at least some truth to these criticisms. My headphone and IEM tuning preferences don’t necessarily align with the Harman Curve. But I’m not sure that they never do, either. That’s because, as Olive has noted, it can be hard to pin down which headphones and IEMs count as Harman-tuned. The Truthear x Crinacle Zero IEM has been lauded for complying with the Harman Curve. On Twitter, Olive confirmed that the Zero is “pretty close to the Harman Target with a little extra bass.” I think that the Zero has bloated bass and sibilant treble. But there are other IEMs that have been described as Harman-tuned that I either like (Moondrop Variations) or love (ThieAudio Monarch MKII).

Why is that? Perhaps claims of Harman compliance get tossed around too loosely. Or maybe there are sonic characteristics beyond frequency response that affect whether I like a Harman Curve-tuned headphone or IEM. I can’t say for sure.

Regardless, I respect Harman and the work that Olive and others have done. A few years ago, I worked my way through the levels in Harman’s excellent How to Listen ear-training software. That led me to Sound Gym and Berklee’s “Critical Listening“ class. Love or hate the Harman Curve, there just aren’t many audio companies conducting publicly available audio research like Harman’s.

Where things go wrong is when reviewers treat the Harman Curve as an absolute truth. The reviewer mentioned above asserted both that the Harman Curve represents scientifically proven “facts, not opinions” and that any headphone or IEM with a different frequency response is “very incorrect.” He even argued that the Harman Curve should become a “universal standard” with which all headphones and IEMs must align:

There is another major reason why what he [the non-Harman-tuned headphone’s designer] is doing is wrong: it continues to randomize the music industry [emphasis in original]. We all need to get behind one standard that is used both in production and playback. Only then we have any hope of hearing what the artist approved (and heard). If we continue this wild west of every designer’s idea being right, we will never get there. With a single standard that is close to what many like, we have a headphone that can be used without EQ. And those that don’t agree, can EQ to what they like.

We have such a target right now: Harman’s. We all need to get behind that if we want to hear music the way it is intended. Anything else helps a company at the expense of us, the consumers and music lovers....

Both the recording and playback industry need to agree on one curve.... With our efforts here highlighting importance of Harman target, we are seeing more and more companies using it. We need more companies to get behind it and work towards a universal standard….

[W]e need to kiss the ground that we can predict listener preference with 60 to 70% probability. If we had a disease and a new medication came to market with this level of efficacy, v[ersu]s others with far less, we would be dancing in streets.

Ethically, this kind of dogmatism is inappropriate for a reviewer. In this case, it’s also fundamentally misleading.

Harman’s preference research is nothing like a pharmaceutical trial, either methodologically or practically. Clinical trials generally involve thousands of participants, many years, tens of millions of dollars, ample controls, and rigorous review by outside experts. For a new medication to gain approval, the trial needs to show that it’s safe and effective, not just for the study’s participants but also for the diverse population at large. For that reason, such studies often take the form of a randomized controlled trial, which is considered the “gold standard“ for medical research. As UNICEF explains, “RCTs typically use both random sampling (since they are usually aiming to make inferences about a larger population) and random assignment (an essential characteristic of an RCT).”

Harman’s research is clever, but it’s decidedly less rigorous than a clinical trial. Some of the company’s preference studies use samples as small as ten people. The largest I could find included 249 listeners in multiple countries. For comparison, Pew’s respected global attitudes survey samples at least 1,000 respondents per country. Large sample sizes increase the likelihood that the research has what’s known as “external validity,” which means that the results can be generalized beyond the study’s participants.

Generalizability also depends upon selecting study participants using “probability sampling,” a “technique involving randomization, so that members of the population or sub-groups within it have known probabilities of inclusion.” The Harman Curve research isn’t based on probability sampling. The 249-person sample was drawn from four Harman offices, four universities, and JBL dealers. This type of sampling is called “convenience sampling,” and it has serious limitations. As is often stated in statistics and survey research articles and textbooks, “You should never attempt to generalize convenience sample results to the population.” It’s possible to use fancy methods to combine a convenience sample with a probability sample, though how well such methods stack up against true probability sampling remains a point of debate among statisticians. More often, researchers with non-probability data try to reach “saturation,” the point at which there’s nothing more to be learned by adding more participants. But deciding when a study has reached saturation is itself a subjective judgment.

That’s not to say that studies based on convenience samples are worthless. Probability sampling is expensive and inconvenient. So plenty of market research and even academic studies use convenience sampling. Peruse almost any psychology journal, and you’ll find convenience samples. It’s common for Psych 101 or Experimental Psych students to be cajoled into participating in a faculty member’s study. Sometimes participation is even required to pass the course. That said, psychology is in the midst of a discipline-roiling replication crisis at least partly because convenience samples have low external validity.

Things get even more complicated when a study uses a subjective rating scale, as is the case with Harman’s preference research (and lots of academic work). Do participants think of the scale’s intervals as equal, where a rating of 10 is twice as good as a rating of five? Or do they think of it as simple ordering? Seemingly small distinctions like those end up mattering a lot when it comes to drawing conclusions from the study’s data.

Unlike some fervent fans of the Harman Curve, Harman’s lead researcher Sean Olive is well-aware of these methodological limitations and others. In almost every article or presentation, he includes disclaimers such as “listener sample was not randomly selected or balanced,” “method of adjustments in bass and treble not loudness compensated: some listeners may have simply boosted bass and treble because it made the music louder,” and “it is important for the reader not to draw generalizations from these results beyond the conditions we tested.” In fact, Olive has written, “There is no single target curve that satisfies all listeners. It’s a statistical exercise[.]” He’s expressed openness to “put[ing] some confidence limits around the curve maybe based on preferred bass and treble levels data,” and recently suggested a “preference bounds” target that features a 6 dB range in the low bass, 2.5 dB range in the midrange, and 7 dB range in the upper treble. He’s (figuratively and literally) embraced critics of the Harman research. He’s even poked fun at audiophiles’ overheated debates about the Harman Curve.

We can’t be sure that the Harman Curve represents what 64 percent of the public or even 64 percent of audiophiles prefer. All we can say for sure is that it’s what 64 percent of the study’s participants liked. None of the Harman studies were particularly gender-balanced. The 249-person sample was 89 percent male and 11 percent female. A 238-person study’s sample was 86 percent male and 14 percent female. A 130-person study’s sample was 78 percent male and 22 percent female. That’s a big problem when women comprise half of the population. It’s an even bigger problem considering that Harman’s research has found significant gender differences in frequency response preference.

At best, Harman’s research suggests that the Harman Curve will appeal to a not-insignificant slice of consumers. That’s undoubtedly helpful to Harman when it designs its own headphones. But that’s very different from the aforementioned reviewer’s assertion that the Harman Curve is a scientific fact that should be imposed upon all manufacturers and consumers.

It’s also important to remember that the Harman Curve represents consumer preferences, not “correct” tonality. While the curve began with a GRAS-equipped dummy listening to the in-room response of good speakers, the study’s participants tweaked the bass and (sometimes) treble to their liking from there. This feedback is what produced the Harman Curve’s elevated bass. According to the reviewer, that’s because “without tactile feedback that speakers bring, headphones need more bass…. [Y]ou have no tactile feedback in headphones. You need that boost to get something to replace that.”

It’s true that when moving from speakers to headphones or from headphone to IEMs, the subjective experience of bass as a simultaneous sound and sensation tilts away from the latter and towards the former. The Harman Curve is not alone in advancing a bass boost is advisable when the physical sensation is lost. Some bookshelf speakers, for example, include a slight bump in the upper bass in an attempt to offset their lack of low bass.

Ultimately, though, each listener will perceive this sensation-sound tradeoff differently. Harman’s own research found that “[t]he effect of vibration on preferred bass level was somewhat dependent on the listener, which could be related to their weight.” A single headphone or IEM frequency response standard like the Harman Curve can’t account for that variation.

We also don’t know whether other companies have carried out listener preference studies similar to Harman’s and arrived at different results. Most businesses prefer to keep their market research proprietary. Even a small company like Schiit uses blind listening to help it decide which products to bring to market. It’s hard to believe that large ones like Sony and Sennheiser are giving their engineers carte blanche to design headphones and IEMs without regard for what consumers like. Yet products like Sony’s IER-Z1R and Sennheiser’s HD820 deviate substantially from the Harman Curve. I reviewed the latter and didn’t like it. But I know that plenty of consumers did, and Sony may have even had internal research that made them confident its tuning would be popular.

Crucially, even if we assume that Harman’s results can be generalized to all consumers, that doesn’t mean that we’d be better off if all headphones and IEMs followed it.

Harman’s research shows that headphones that deviate from the Harman Curve can be just as popular ones that follow it. In a 2018 study, Olive and his coauthors pitted 31 headphones against the Harman Curve. The study’s sample was composed of 130 Harman employees, 102 of whom “were tested for normal audiometric hearing and had successfully completed level eight for all tasks in the training software ‘Harman How to Listen.’” It found that the Harman Curve was tied for most preferred with three other headphones. More recent research —conducted by other researchers and analyzed by Olive — showed that 11 frequency responses, including Sound Guys’ universal target and Harman’s 2018 over-ear target, were tied for most preferred. In Olive’s brand new research, Harman’s 2019 in-ear target and Sound Guys’ target were tied for most preferred.

Notably, Sound Guys’ target has significantly less energy below 100 Hz and between 2 and 6 kHz than all versions of the Harman curve, especially the 2019 in-ear target.

To state the obvious, even if it’s true that 64 percent of listeners prefer the Harman Curve, 64 is a long way from 100. Somewhere between 550 million and 1.2 billion headphones are sold each year. While some people, including yours truly, are buying multiple headphones per year, 36 percent of headphone consumers is a whole lot of people. If there weren’t plenty of people who prefer non-Harman frequency responses, companies would stop making products that deviate from the Harman Curve. Isn’t it good that companies are meeting those consumer’s tastes?

The reviewer in question says no. He argues that those consumers’ preferences have “no value” because they “don’t have critical listening abilities” and “will buy anything that... looks nice.” In other words, he thinks that the 36 percent of listeners who enjoy a non-Harman frequency response are a bunch of rubes. This kind of insulting reviewing is becoming all too common and should be rejected, regardless of the underlying opinion. When a popular YouTube reviewer calls those who like the Truthear x Crinacle Zero “complete idiots who know absolutely NOTHING about what real sound quality is,” it hardly matters that I don’t like the Zero, either. Such condescension is abhorrent.

Once you believe that everyone who disagrees with your tastes is a nincompoop, it’s easy to leap to the conclusion that their freedom of choice must be revoked in order to save them from their base impulses. According to the Harman-loving reviewer, “You as a consumer should be all in favor of standardization.” He argues that headphone and IEM buyers need to have a single frequency response imposed upon them “[f]or the same reason there are not 1,000 flavors at [the] [i]ce [c]ream store.” Putting aside the fact that Baskin Robbins has “more than 1,400 [flavors] in its flavor library” and flavor choice is seen as a selling point for ice cream parlors, it’s worth asking whether headphone buyers are worse off because they’re able to choose among headphones and IEMs with varying frequency responses.

The reviewer’s argument goes against the “customer is always right” mentality of modern American capitalism. I’m actually sympathetic to criticisms of that mentality. Yet it’s hard to see how those criticisms apply to headphones or IEMs. Some research that suggests that too many choices increase consumers’ anxiety and unhappiness, but that research has been ensnared by psychology’s replication crisis and is far from universally accepted.

The strongest arguments against consumer choice involve issues of public policy and social, rather than consumer, welfare. There’s good evidence that, in areas such as healthcare and education, choice actually makes everyone worse off. In those contexts, private choice can undermine the “positive externalities” (social benefits) created by public goods.

For consumer products like headphones or ice cream, though, variety is almost always good. Research has found that “as new products are introduced, households can choose consumption bundles better suited to their particular tastes, resulting in welfare gains from better selection.” According to one study, “welfare gain from increased variety was about 40 times the gain from lower prices,” in part because “demand for these niche products was significantly more inelastic than that of mass products.” In layperson’s terms, this means that product variety increases consumer welfare precisely because people with less-common tastes strongly value the ability to buy products that match their tastes.

Whatever hypothetical benefits might accrue from mandating that headphones and IEMs follow the Harman Curve are likely to be far outweighed by the welfare loss of denying those who don’t like it the opportunity to buy products they enjoy. The reviewer seems to understand this on some level when he suggests that “those that don’t agree [with the Harman Curve] can EQ to what they like.” But that’s not a reasonable concession. Should 36 percent of consumers be forced to pay a financial and/or convenience tax because they don’t like the Herman Curve? What about consumers who can’t (or don’t want to) EQ their headphones and IEMs? Plus, if EQ’ing is as frictionless as the pro-Harman reviewer implies, then there’s no need to mandate the Harman Curve. Anyone who likes the curve can simply EQ their headphones and IEMs to match it.

The only sure winner if the Harman Curve were made the industry standard would be the Harman company. Sales of Harman’s own curve-aligned headphones and IEMs would almost certainly increase. Harman also could create a “Harman Curve Certified” program for other companies’ products. Audiophiles have seen this before with certifications like MFi, Hi-Res Audio, HDMI, and MQA. These schemes are great for the companies that run them, but they drive up costs for consumers.

Is Harman trying to do this with the Harman Curve? Despite some Harman-lovers’ wishes to the contrary, I haven’t seen any evidence suggesting that Harman wants to make the Harman Curve the industry standard. In a recent presentation titled “Will the Headphone Industry Ever Agree on a Design Target Curve?,” Olive wrote, “While there is a strong argument for a better industry standard headphone target, loudspeaker history tells us it is unlikely to occur.”

Even if Harman has no intention of turning the Harman Curve into a standard, it’s important to remember that the company’s decision to make the Harman Curve research public wasn’t purely altruistic. Harman has kept other research private. By publicizing the Harman Curve, the company can use its research to promote both its products and its technical and consulting services. For example, Harman advertises its AKG K371 headphones as “meticulously engineered to match AKG’s Reference Response Curve, the result of an extensive 5-year study by HARMAN that tested hundreds of subjects on how they perceive audio, and more importantly, what their preferences are for critical listening. This new benchmark of accuracy ensures that K371 headphones provide accurate, neutral sound.”

There’s nothing wrong with that. Claiming that your products are by “science” is a time-tested marketing strategy. Such claims might persuade consumers to buy that company’s products, or they could backfire. But — and here is where things get really interesting — if the science in question measures consumer preferences, then a successful marketing campaign centered on the science actually creates further support for the science.

If Harman’s marketing convinces consumers that the Harman Curve is scientifically accurate, then consumers might buy more Harman-tuned headphones and IEMs. As the pro-Harman reviewer noted, “The current high-end headphone market is heavily influenced by what people read and watch on [Y]ou[T]ube. People can be convinced anything is good and a business made around it.” As more and more consumers buy Harman Curve-aligned products, then that’s the frequency response they’ll become accustomed to. Quoting the reviewer again, “If you have been listening to this headphone for a while, your ear/brain have adapted so its tonality will seem right to you and anything else wrong. This is why many people like even broken speakers/headphones.” So if Harman’s curve-touting marketing works, it could influence the results of future preference studies, thereby creating its own proof that the Harman Curve is “correct.”

Indeed, other brands’ past marketing success may have influenced the development of the Harman Curve. Harman’s research found that younger listeners prefer much more bass that older listeners. One common hypothesis for why this is the case is that the popularity of Beats headphones among young consumers has conditioned them to want more bass. That’s precisely why consumer preference studies can only prove that something is popular, not that it’s “scientifically correct.”

Some reviewers take pains to make this clear to readers. RTings recently updated its procedures for headphone reviews and measurements, noting:

Tests related to the headphones’ frequency response, like bass, mid, and treble accuracy, are based on (and scored against) our target curve. However, their current state may not give a good picture of whether you’ll like their sound. We’re not getting rid of objective measurements, though! Instead, we want to complement the data by clarifying the possibility of different interpretations depending on your preferences, which may or may not also align with our target. For example, if you like a more bass-heavy sound, our target curve framework may not make it easy to find what you’re looking for…. It’s paramount to detangle our preference-based results from our scores. We can provide information on what most users like; however, it’s also important to remember that sound quality is highly subjective and that there isn’t a unanimous way to evaluate it.

Reviewers who love the Harman Curve are free to praise it and to withhold positive reviews from headphones and IEMs that don’t follow it. But it’s unethical to insist that it’s the only scientifically valid frequency response. It’s condescending to call consumers who dislike the curve (or don’t like it exclusively) fools. And it’s misleading (at best) to pretend that headphone and IEM consumers will be better off if the Harman Curve is their only option.