AES and Its Papers

[The Computer Audiophile]

Hi Guys, recently the topic of the Audio Engineering Society came up in the MQA thread. I was contacted by a representative of the AES because s/he claimed what was said was untrue. Seeking to correct the record, I inquired more about the AES. While the conversation went off the rails, delved into MQA, and didn't end how I wish it would've, I gleaned a little bit of information from the exchange. 

 

Before delving into this further, I want to make sure people know that I'm not out to get anyone, including the AES. I benefit greatly from the AES, its members, and readers. Without the engineers who publish and read the AES, my enjoyment of this wonderful music listening hobby would be diminished. I only want to help educate people about the AES and present facts.

 

By starting this separate topic, I'd love for this to be a discussion about the AES and its papers, and how these are used with respect to our side of the world as consumers. I hope people can chime in with facts or personal experience about the topic. Hyperbole and misrepresentations aren't helpful and aren't allowed. 

 

Backstory:

 

In the MQA thread, it was mentioned that the AES isn't a scientific publication etc... Here's a link and I'll quote the text below for context. 

 

On 12/28/2020 at 10:04 PM, botrytis said:

AES is not a scientific publication. True scientific publications are 'Peer reviewed'. This means that experts in the field look at the data supplied and determine it is it REAL or not. AES is an advert publication. Meaning blow your own horn w/o anyone looking over your shoulder.

 

I responded by saying:

 

On 12/28/2020 at 10:14 PM, The Computer Audiophile said:

This can’t be stressed enough.

 

In fact, the AES as an organization is much more political than its fans would like to believe. 

 

 

The AES representative took issue with the original post and my response. I won't publish the text of our email exchange, as the representative asked it remain private unfortunately. If s/he is willing to allow publication of the transcript I would be thrilled to publish it as it would provide more transparency on this topic. 

 

That said, let's discuss AES publications and its peer review process. The reason why I think this is very valuable for us as consumers is that these AES publications are frequently used as badges of honor by engineers, companies, and manufacturers to tout technologies and products to consumers. We often see non-industry fans of certain technologies reference AES publications in support of one side of an argument. 

 

Based on my email exchange and Internet research, it appears that AES has Convention Papers (not published in the official Journal) and the following five other types of writing that are published in its Journal (Research Papers, Engineering Reports, Review Papers, Communications, and Letters to the Editor). According to the AES, with resect to the Journal writings, "All submissions will go through a peer review process to check their suitability for JAES." I'll touch on that in a bit. 

 

My takeaway from this is that consumers must differentiate between Convention Papers that aren't peer reviewed prior to publication and those in the journal that are peer reviewed. This may be valuable information when we are given an AES "Paper" as evidence why an engineer's technology has efficacy or why a product is superior etc... Sources matter. 

 

When looking into the peer review process the AES uses I was disheartened to learn that it's quite opaque. This is of course my opinion and I realize this type of opacity is used in other professions, but I believe more transparency would benefit everyone, especially consumers as we are almost always who receives the marketing push for the products mentioned in the paper. 

 

Here is what I found.

 

1. Peer reviewers are always anonymous and never known to the authors of the papers.

2. Peers are industry experts and professors that are aware of standards in their fields and standards for journal publication.

 

 

Number 2 above is great, I believe. A mix of professors and industry experts are needed. The best engineers are frequently in both fields. As engineers like to say, those that can, do, and those that can't, write. The best engineers are hired by the best companies and universities. All good there. 

 

Number 1 is an issue that isn't specific to the AES, but is an issue nonetheless. Without a view into who actually reviewed something, the consumer has no clue about bias, agendas, or even skills of the peer doing the review. The AES told me it favors this anonymous approach because there are a limited number of qualified researchers and they usually know each other. Anonymity avoids hurting personal relationships and reputations when something is criticized. To me, sources are everything. This opaque approach favors personal feelings over transparency and objectivity. Again, just my opinion because I struggle to accept statements from Ministers of Information. 

 

I will use one small portion of the email I received from the AES representative below. This speaks for itself, as to what the AES thinks of consumers who are the only reason, in the end, the publication exists. Without consumers, nobody is going to invent 90% of the stuff mentioned in the Journal. OK, I made up the 90% number, because I'm sure there are valid medical advances in instrumentation etc.. but again, consumers or patients are the reason this stuff is invented. 

 

"JAES and professional journals in general are academic publications, written to a professional audience and not to consumers.  They have no obligation at all to provide reviewer data to people who have no ability to assess it, much less make informed decisions about “political bias” — good grief!"

 

Think of this statement what you will, but I believe consumers have the ability to assess agendas, biases, and other items that may impact a peer reviewed paper. Nobody believes consumers can digest the efficacy of the  information  inside the paper, but consumers certainly can make judgements about the peers who reviewed the paper. 

 

OK, that's what I've found, mixed with my take on it all. Please chime in with facts or personal experience that may help consumers understand what's being fed to them, when they evaluate a product. 

[The Computer Audiophile]

In the context of how this whole topic started, MQA, it's difficult to replicate the results given the patent application. Also, the MQA AES paper was just a convention paper without any peer review. However, it's usually discussed as an AES Paper, not Convention Paper. 

 

Edit: I was directed to the MQA AES Paper by the person I've been exchanging emails with form AES. This came out roughly five years after the Convention Paper - https://www.aes.org/e-lib/browse.cfm?elib=20456

[The Computer Audiophile]

Great information on how these things work. 

 

I still don't like it. Putting one's name on something is the ultimate in transparency and honesty. In fact, I see some people advocating for use of real names on the internet because anonymous people behind keyboards can be a bad thing. 

[The Computer Audiophile]

Just now, Solstice380 said:

Then one of the authors of a cold fusion paper comes and shoots you for calling out their shoddy work.  No thanks!  (although those reviewers really missed that one!)

You'll never stop crazy :~)

[The Computer Audiophile]

16 minutes ago, andrewinukm said:

Accusing before seeking to understand the process is a common flaw. I am in full support of information transparency.

 

However, there are many considerations when deciding what level of transparency. Information transparency does not guarantee non-bias, it only means more information is available. More information could be a good thing, or it could make things worse. 

 

The reviewers didn't even write the papers, and here we are, arguing if the reviewers names should be published. Then where do we stop? Should they publish the names of lecturers who marked the reviewers' test papers in college? 

 

If a person is a subject matter expert, the reviewers will matter little. This person would be able to assess the quality and validity of the paper using his expertise. For the layman, any form of information within the paper is prone to be misinterpreted anyway. Declaring the reviewers only means one more thing to be misinterpreted, while it does nothing to reduce bias in the scientific publication process.

 

 

Reviewers give their stamp of approval, which equates to a very powerful message for those who aren’t learned in a subject. Because peer reviewed papers are often used as badges of honor toward consumers, we should know who stamped it. If the process was so sound, it should be even better for each paper if we know that touted engineer Joe Public approves it. On the other hand, if the only people available to review a paper were far less than experts, but a paper needed to get published, so they went through with it, people should know. 
 

Transparency builds trust. Opacity builds skepticism. 
 

In my view anonymity is the old school minister of information style. The trust us because we know better style doesn’t build confidence. 

[The Computer Audiophile]

5 minutes ago, Kal Rubinson said:

In a scientific journal, the role of the reviewer is to insure that the work was done properly but there is no stamp of approval on the results or the conclusions.

 

The audience for peer-reviewed papers is not the consumer.  It is the relevant scientific community.

 

Very likely.  

I certainly hear you Kal, but I can’t count how many times I’ve received a marketing pitch that included an AES paper mention to bolster credibility. 
 

If all of this stayed in the domain of the scientific community, it would be fine. It’s the crossover for marketing that causes issues.