AES and Its Papers

[The Computer Audiophile]

Hi Guys, recently the topic of the Audio Engineering Society came up in the MQA thread. I was contacted by a representative of the AES because s/he claimed what was said was untrue. Seeking to correct the record, I inquired more about the AES. While the conversation went off the rails, delved into MQA, and didn't end how I wish it would've, I gleaned a little bit of information from the exchange. 

 

Before delving into this further, I want to make sure people know that I'm not out to get anyone, including the AES. I benefit greatly from the AES, its members, and readers. Without the engineers who publish and read the AES, my enjoyment of this wonderful music listening hobby would be diminished. I only want to help educate people about the AES and present facts.

 

By starting this separate topic, I'd love for this to be a discussion about the AES and its papers, and how these are used with respect to our side of the world as consumers. I hope people can chime in with facts or personal experience about the topic. Hyperbole and misrepresentations aren't helpful and aren't allowed. 

 

Backstory:

 

In the MQA thread, it was mentioned that the AES isn't a scientific publication etc... Here's a link and I'll quote the text below for context. 

 

On 12/28/2020 at 10:04 PM, botrytis said:

AES is not a scientific publication. True scientific publications are 'Peer reviewed'. This means that experts in the field look at the data supplied and determine it is it REAL or not. AES is an advert publication. Meaning blow your own horn w/o anyone looking over your shoulder.

 

I responded by saying:

 

On 12/28/2020 at 10:14 PM, The Computer Audiophile said:

This can’t be stressed enough.

 

In fact, the AES as an organization is much more political than its fans would like to believe. 

 

 

The AES representative took issue with the original post and my response. I won't publish the text of our email exchange, as the representative asked it remain private unfortunately. If s/he is willing to allow publication of the transcript I would be thrilled to publish it as it would provide more transparency on this topic. 

 

That said, let's discuss AES publications and its peer review process. The reason why I think this is very valuable for us as consumers is that these AES publications are frequently used as badges of honor by engineers, companies, and manufacturers to tout technologies and products to consumers. We often see non-industry fans of certain technologies reference AES publications in support of one side of an argument. 

 

Based on my email exchange and Internet research, it appears that AES has Convention Papers (not published in the official Journal) and the following five other types of writing that are published in its Journal (Research Papers, Engineering Reports, Review Papers, Communications, and Letters to the Editor). According to the AES, with resect to the Journal writings, "All submissions will go through a peer review process to check their suitability for JAES." I'll touch on that in a bit. 

 

My takeaway from this is that consumers must differentiate between Convention Papers that aren't peer reviewed prior to publication and those in the journal that are peer reviewed. This may be valuable information when we are given an AES "Paper" as evidence why an engineer's technology has efficacy or why a product is superior etc... Sources matter. 

 

When looking into the peer review process the AES uses I was disheartened to learn that it's quite opaque. This is of course my opinion and I realize this type of opacity is used in other professions, but I believe more transparency would benefit everyone, especially consumers as we are almost always who receives the marketing push for the products mentioned in the paper. 

 

Here is what I found.

 

1. Peer reviewers are always anonymous and never known to the authors of the papers.

2. Peers are industry experts and professors that are aware of standards in their fields and standards for journal publication.

 

 

Number 2 above is great, I believe. A mix of professors and industry experts are needed. The best engineers are frequently in both fields. As engineers like to say, those that can, do, and those that can't, write. The best engineers are hired by the best companies and universities. All good there. 

 

Number 1 is an issue that isn't specific to the AES, but is an issue nonetheless. Without a view into who actually reviewed something, the consumer has no clue about bias, agendas, or even skills of the peer doing the review. The AES told me it favors this anonymous approach because there are a limited number of qualified researchers and they usually know each other. Anonymity avoids hurting personal relationships and reputations when something is criticized. To me, sources are everything. This opaque approach favors personal feelings over transparency and objectivity. Again, just my opinion because I struggle to accept statements from Ministers of Information. 

 

I will use one small portion of the email I received from the AES representative below. This speaks for itself, as to what the AES thinks of consumers who are the only reason, in the end, the publication exists. Without consumers, nobody is going to invent 90% of the stuff mentioned in the Journal. OK, I made up the 90% number, because I'm sure there are valid medical advances in instrumentation etc.. but again, consumers or patients are the reason this stuff is invented. 

 

"JAES and professional journals in general are academic publications, written to a professional audience and not to consumers.  They have no obligation at all to provide reviewer data to people who have no ability to assess it, much less make informed decisions about “political bias” — good grief!"

 

Think of this statement what you will, but I believe consumers have the ability to assess agendas, biases, and other items that may impact a peer reviewed paper. Nobody believes consumers can digest the efficacy of the  information  inside the paper, but consumers certainly can make judgements about the peers who reviewed the paper. 

 

OK, that's what I've found, mixed with my take on it all. Please chime in with facts or personal experience that may help consumers understand what's being fed to them, when they evaluate a product. 

[botrytis]

WOW - I have been involved with peer review process, in another field of study, but I think it bears talking about as in response to this.

 

When I was a post doc, at the University of Georgia, I was asked to review papers for several journals. During this process, which was from my mentor at the school, I gave my reviews of the papers I was asked to review.  I was also told, that my reviews were too harsh because they were friends of my mentor and he just wanted them passed through. I would not do that and was put in the dog house due to that ( very apropos for UGA - 'Mericans will get the pun there). THIS IS ONE OF THE ISSUES WITH PEER REVIEW, friendships should not matter but they do. Others, only allow members to peer review. They do not send it out of the organization. Again, this is common.

 

There is good points and bad points to this process. I am just pointing out some flaws.

[DuckToller]

It sounds to me like the derogative term  "not peer reviewed" indicates (for many not involved within the scientific environment) that "peer reviewed" carries intrinsically the knighthood & highest order of trust level for available scientific objectivity ... and forgoes to review the many different processes behind that term. Given the position of AES you described above, they may not be ultimately unhappy about that ...
Given how many money changes annually from industry to source academic research a peer reviewed paper receives its value imho from the credibility of the peer and not genuinly from the journal it is published in .
.. i could be wrong with this biased view, pls challenge ...

[Kal Rubinson]

1 hour ago, The Computer Audiophile said:

1. Number 1 is an issue that isn't specific to the AES, but is an issue nonetheless. Without a view into who actually reviewed something, the consumer has no clue about bias, agendas, or even skills of the peer doing the review.

 

2.The AES told me it favors this anonymous approach because there are a limited number of qualified researchers and they usually know each other. Anonymity avoids hurting personal relationships and reputations when something is criticized.

 

3.To me, sources are everything. This opaque approach favors personal feelings over transparency and objectivity. Again, just my opinion because I struggle to accept statements from Ministers of Information. 

I added the number for reference:

1. This is the most common paradigm in scientific publications.  I have been on both sides of the matter as author of my own papers and as reviewer of those by others.  The anonymity is essential but has nothing to do with the consumer.  It simply is intended to minimize bias.  In a case where the author believes that a review is biased, they can request another as replacement or to rebalance and it is the editor's responsibility to manage this fairly.

2.  As I mentioned above, that is true.  There were several times that it was clear to me from the text exactly who was my reviewer but, generally, it was not.  As an author, it was really of no consequence but entertaining to speculate. 

3.  A responsible journal lists its editorial board and publicly thanks its reviewers without explicitly linking them to specific papers.  No system is perfect but, ultimately, is judged by the weight of the respect it earns or doesn't and that is hardly a secret.  

4. (Not above.)  The real check on the value of a report is whether it is replicated.  The most significant and interesting findings are usually replicated because they inspire great interest (or great doubt).    

[pkane2001]

20 minutes ago, Kal Rubinson said:

4. (Not above.)  The real check on the value of a report is whether it is replicated.  The most significant and interesting findings are usually replicated because they inspire great interest (or great doubt).    

 

#4 is key. Peer reviews are no guarantee of validity, they are just a safety check to stop mistakes or obvious fraud from being published. Any complex study, especially involving multiple test subjects, can only be reviewed for structural, logical, and mathematical errors. The actual results of such a study need to be replicated to be verified.

[botrytis]

1 hour ago, DuckToller said:

It sounds to me like the derogative term  "not peer reviewed" indicates (for many not involved within the scientific environment) that "peer reviewed" carries intrinsically the knighthood & highest order of trust level for available scientific objectivity ... and forgoes to review the many different processes behind that term. Given the position of AES you described above, they may not be ultimately unhappy about that ...
Given how many money changes annually from industry to source academic research a peer reviewed paper receives its value imho from the credibility of the peer and not genuinly from the journal it is published in .
.. i could be wrong with this biased view, pls challenge ...

 

Well, non peer reviewed journals are like Biofuels - Biofuel Research Journal (biofueljournal.com) Not saying they are bad but they are more like commercials rather than actual scientific papers.

[MarkusBarkus]

43 minutes ago, pkane2001 said:

Any complex study, especially involving multiple test subjects, can only be reviewed for structural, logical, and mathematical errors. The actual results of such a study need to be replicated to be verified.

It's true the replication of experiments and studies are critical; however, most of the articles and findings which are published (any field) are not large studies (like the Covid19 studies in the news), they are smaller findings describing mechanisms, structures, or behaviors (grossly over simplified list).

 

These could be macro (cosmic) or micro (protein folding, binding sites, crystal structures, etc.). The integrity of the findings is critical because they are often foundational for advancing other work, or validating investigations others are working on.  

Sometimes submitting authors have an idea who is reviewing the papers. Some journals you can request reviewers (not guaranteed). Typically, there are three reviewers. Usually one is out of their depth.

 

They refute findings, and/or require experiments to be completed to support the author's assertions. It often takes many months, a year, to satisfy reviewers and editors before publication.

 

Peer review may not be "knighthood" but it's the best process IMO, we have. And there is absolutely a hierarchy of journals, impact factor ratings, etc. Some journals take almost anything. And scientists know who they are.
 

The existence of two distinct "avenues" for publication is probably a good source of confusion (best case) or exaggerated/unsubstantiated claims (or worse). 
 

This similarly surfaced during the Covid19 information overload. Journos were reading pre-publication results, not yet peer reviewed, and publishing it as agreed-upon science, or "Science" upper case. Whether they even understood what was pre-published is another matter.

 

The non-peer reviewed articles are just informational/interesting/food for thought. It ain't hard-science.
 

Most of what we read about hifi stuff is "just" a white paper. Not meant as derogatory. 
 

The hifi product model is interesting, because the customers are kind of the study participants. Did I wander OT? Sorry.

[The Computer Audiophile]

In the context of how this whole topic started, MQA, it's difficult to replicate the results given the patent application. Also, the MQA AES paper was just a convention paper without any peer review. However, it's usually discussed as an AES Paper, not Convention Paper. 

 

Edit: I was directed to the MQA AES Paper by the person I've been exchanging emails with form AES. This came out roughly five years after the Convention Paper - https://www.aes.org/e-lib/browse.cfm?elib=20456

[Allan F]

3 hours ago, The Computer Audiophile said:

Number 2 above is great, I believe. A mix of professors and industry experts are needed. The best engineers are frequently in both fields. As engineers like to say, those that can, do, and those that can't, write. The best engineers are hired by the best companies and universities. All good there. 

 

Number 1 is an issue that isn't specific to the AES, but is an issue nonetheless. Without a view into who actually reviewed something, the consumer has no clue about bias, agendas, or even skills of the peer doing the review.

It is all well and good that "peers are industry experts and professors that are aware of standards in their fields and standards for journal publication". Of greater importance, however, is whether they adhere to those standards. While publishing the names of peer reviewers does not necessarily provide a basis for consumers to assess this, it does provide insight into possible biases, etc. that may be relevant.

[fas42]

What amuses me when I come across some of the AES papers, peer reviewed or otherwise, are the massive assumptions they frequently contain - "We used an acceptable quality replay system to test this; therefore, the results are valid" type of thing ...

[KeenObserver]

This is, indeed, an excellent topic for discussion.

[MarkusBarkus]

37 minutes ago, Allan F said:

It is all well and good that "peers are industry experts and professors that are aware of standards in their fields and standards for journal publication". Of greater importance, however, is whether they adhere to those standards.

...often, the peer reviewers are competitors, actually. And there have been cases of reviewers beginning "new" areas of research based on something they "reviewed." 
 

That said, IMO generally speaking, folks seem collegial and respectful. And actually, the better/higher the science, the fewer people exist--on the planet--to review it. 
 

In this regard, I guess I would echo the unnamed email respondent, most folks would not be able to evaluate or even understand the bleeding edge stuff--or maybe even more "ordinary" publications. Some, of course, would be.
 

It's not a publishing cabal of reviewers and editors, IMO, although there must be a greater-than-zero possibility that "favors" are done.

 

It is not a perfect system, by any means, but the anonymity is intended to prevent grudges, rather than helping buddies out. It usually works out OK, but it is very slow. And...it's possible to get scooped.

 

Just by way of example: Here is my favorite article title from one of last week's journals around the house: "Galactosaminogalactan Activates the Inflammasome to Provide Host Protection." Not engineering, I fully understand, but my point is it would be challenging, IMO, to find many folks who could usefully criticize the contents or the motivations of the reviewers or editors. It seems like we could, but I don't think we actually could, in most cases. 

 

BTW re: above article:

Received 3/4/20

Accepted 9/23/20

Published Online 12/2/20

 

I think @ The Computer Audiophile identifying for us that there are two paths for publication is very useful. As @botrytiswrote, the non-peer reviewed stuff is essentially marketing, or "this looks interesting" level stuff.

[Allan F]

13 minutes ago, MarkusBarkus said:

It is not a perfect system, by any means, but the anonymity is intended to prevent grudges, rather than helping buddies out. It usually works out OK, but it is very slow. And...it's possible to get scooped.

 

How is AES different from other respected journals that publish the names of peer reviewers without these apparent "grudges" coming into play? If egos are that fragile, perhaps the selection process of these peer reviewers requires re-examination.

[botrytis]

2 hours ago, Allan F said:

How is AES different from other respected journals that publish the names of peer reviewers without these apparent "grudges" coming into play? If egos are that fragile, perhaps the selection process of these peer reviewers requires re-examination.

OK. first is a conference paper (Batelle paper) - they are usually based on presentations at a conference or a poster and are only peer reviewed before they are accepted (on the abstract sent to the conference - 99% are accepted if they are on topic).

 

The second is a peer reviewed paper - notice the difference in style. Also notice the time from submission to publication - almost a year and that is common since there is always revisions and the paper is reviewed again.

 

With most professional journals, one does not need to be a member to publish in them. If you look at the RS and PC paper, them being mentioned as lifetime members of AES gives me pause - SO WHAT? Why is that ON the paper at all?

 

Battelle_03MTfinalWDH351.pdf RS_Laccase1996.pdf

[R1200CL]

3 hours ago, The Computer Audiophile said:

In the context of how this whole topic started, MQA, it's difficult to replicate the results given the patent application. Also, the MQA AES paper was just a convention paper without any peer review. However, it's usually discussed as an AES Paper, not Convention Paper. 

 

Edit: I was directed to the MQA AES Paper by the person I've been exchanging emails with form AES. This came out roughly five years after the Convention Paper - https://www.aes.org/e-lib/browse.cfm?elib=20456

I wonder what happens if one actually started a discussion about MQA here 😀

 

[botrytis]

Now, the reason I called the BS article an advertisement is the back page of the article. In opinions, letters to the editor, etc. one sees the background of the authors listed. In regular scientific papers. one does not see that. The science should stand alone, the person's background should not be important to the content of the paper.

 

 

[Kal Rubinson]

32 minutes ago, botrytis said:

Now, the reason I called the BS article an advertisement is the back page of the article. In opinions, letters to the editor, etc. one sees the background of the authors listed. In regular scientific papers. one does not see that. The science should stand alone, the person's background should not be important to the content of the paper.

Did it have links to their Facebook pages? 😁

 

P.S.:  I remember the great Wazoo.  Really nice amp.

[botrytis]

6 minutes ago, Kal Rubinson said:

Did it have links to their Facebook pages? 😁

 

P.S.:  I remember the great Wazoo.  Really nice amp.

 

What's a Facebook?

[NOMBEDES]

Replication of findings is most difficult in the case of audio hardware or software. How can you state with any certainty that amplifier X or cable Y will sound the same when it is transported from one environment to another?

 

In the vast wasteland of audio journalism the only constant is that good recordings sound, well...good and poor recordings sound not so good.  Some audio enthusiasts like  MQA some don’t.  Trust your own ears, no one can help you, not the reviewer, not the salesperson and certainly not the AES.

 

You may spend the equivalent of the GNP of a failed third world state (like Texas) and still not enjoy a poor recording, MQA or not.

[lucretius]

9 hours ago, Kal Rubinson said:

4. (Not above.)  The real check on the value of a report is whether it is replicated.  The most significant and interesting findings are usually replicated because they inspire great interest (or great doubt). 

 

I guess you are referring to seminal works, sometimes called pivotal or landmark studies.  These articles initially presented an idea of great importance or influence within a particular discipline. Seminal articles are referred to time and time again in the research, so you are likely to see these sources frequently cited in other journal articles, books, dissertations, etc.

 

[Kal Rubinson]

8 hours ago, NOMBEDES said:

Replication of findings is most difficult in the case of audio hardware or software. How can you state with any certainty that amplifier X or cable Y will sound the same when it is transported from one environment to another?

We are talking about the engineering and, perhaps, the science behind audio hardware and software.  In that context, your question is not relevant.  It lies in the area of psychoacoustics.

 

[Kal Rubinson]

7 hours ago, lucretius said:

I guess you are referring to seminal works, sometimes called pivotal or landmark studies.  These articles initially presented an idea of great importance or influence within a particular discipline. Seminal articles are referred to time and time again in the research, so you are likely to see these sources frequently cited in other journal articles, books, dissertations, etc.

Certainly those but, also, any non-trivial finding.

[NOMBEDES]

30 minutes ago, Kal Rubinson said:

We are talking about the engineering and, perhaps, the science behind audio hardware and software.  In that context, your question is not relevant.  It lies in the area of psychoacoustics.

 

 

Yes.  I understand.  
 

But with all our engineering and science can we overcome poor recording practices or the quirks of our home environment?
 

 

[botrytis]

1 hour ago, NOMBEDES said:

 

Yes.  I understand.  
 

But with all our engineering and science can we overcome poor recording practices or the quirks of our home environment?
 

 

 

I think it can but, we need to separate the dust from the diamonds. All we can do is setup the best system/room for the money we can afford. We cannot control what others do but we can understand it and thereby explain what is happening. It is complex but not unexplainable. 

 

Some practices might never change does that mean we need to give up? Probably not.

[jabbr]

22 hours ago, The Computer Audiophile said:

In the context of how this whole topic started, MQA, it's difficult to replicate the results given the patent application. Also, the MQA AES paper was just a convention paper without any peer review. However, it's usually discussed as an AES Paper, not Convention Paper. 

 

Edit: I was directed to the MQA AES Paper by the person I've been exchanging emails with form AES. This came out roughly five years after the Convention Paper - https://www.aes.org/e-lib/browse.cfm?elib=20456

 

This is a well known "trick" when giving a lecture at a conference, always write up the talk as a paper because when it is published, it will get accepted without much if any real peer review. Don't believe every scientific paper as the gospel truth. If 10% of the papers published in Science and Nature every week were breakthroughs then humanity would be a lot collectively smarter than we are today.

 

The bottom line is that you need to know the biases of the author when reading anything including a published scientific paper. We need independent journalists to keep those biases in check by exposing them.