AES and Its Papers

[fas42]

8 hours ago, NOMBEDES said:

 

Yes.  I understand.  
 

But with all our engineering and science can we overcome poor recording practices or the quirks of our home environment?
 

 

 

Yes, one can. Turns out that the human hearing system is remarkably capable of compensating for deficiencies in what it listens to - completely unconsciously. However, it still has limitations - when two conflicting sources of degrading distortion are overlaid onto the original captured event, then it's too much - result is fatiguing listening, or it being just plain too unpleasant or annoying to suffer for any length of time.

 

So, the options are: brilliantly recorded music played on mediocre and above systems - this works; and the other being recordings at all ranges of quality replayed on a setup which adds zero subjectively audible artifacts - this also works. The latter is somewhat hard to achieve, but worth it - because it opens one up to being able to enjoy the vast legacy of a century plus of recording ...

[bluesman]

This is an explanation and defense of the peer review process followed by JAES and hundreds of top medical and scientific journals.  It is not in any way a defense of AES, which I left about 20 years ago after being a member for about that long. They do it the same way it's done by every top publication and specialty society with which I'm associated, including JAMA, NEJM etc.  I've been the editor of 2, associate or section editor of 5, and reviewer for 15 or so of the world's top medical journals over the last 40 years, and JAES' PR guidelines are identical to those of the best of the best.  Here's a link to the AES submission review process description, which clearly describes the fact that papers submitted for presentation are subjected to the same review process used for pure journal submissions.  Unless this has been revised since I left the AES, it's still in effect.

 

For virtually all top quality scientific journals, reviewer group members are (as stated in another post) annually thanked and identified along with publication of annual submission / review / acceptance stats.  For example, here's the 2019 version in the Journal of the American Medical Association with publication stats, and here's the actual reviewer list (access is usually for members only, but these links should open a publicly accessible version made available by JAMA). But the identity of reviewers of specific submissions is widely kept confidential, for reasons I'm about to discuss

 

The two major sets of recommendations and guidelines for top quality scientific journalism are the "Uniform requirements for manuscripts submitted to biomedical journals" and the "Best Practice and Guidance" (guidelines, guidance, and other documents) of the Committee on Publication Ethics.  Here's a link to their Ethical Guidelines for Peer Reviewers, which clearly state on page 4 of 12 that peer reviews are to be anonymous for pre-publication review ("Peer reviews are published but not signed").  I wrote the guidelines for publication at the university in which I'm still a professor.  We have a full range of basic science programs (undergraduate, Masters, PhD, fellowships and postdoc) as well as a medical school.  I've researched this intensively, and I know of no authority that recommends open disclosure of the pre-publication reviewers of specific papers.

 

Anonymity is an intergral part of many processes that seriously affect us every day but are totally transparent to most of you, e.g. IRBs (Institutional Review Boards) that approve and oversee clinical research trials, grand jury deliberations that determine whether or not an accused person will be indicted, etc.  The reasons for this are manifold and have been agreed upon by countless people whose accomplishments and reputations are sterling.  Both sides of the coin are represented, in that peer reviewing offers opportunity for retribution, IP theft, inappropriate support and promotion, competitive interference etc - in both directions.  Authors are identified to potential reviewers in order to identify potential COI.  And although this is admittedly on an "honor system" basis, the senior editorial staff of reputable scientific journals are generally quite familiar with their reviewers, their interests, and areas of active research and collaboration. The initial choice of potential reviewers is guided by this knowledge.

 

Does it always work well?  Of course it doesn't.  Some reviewers only blow their own horns, some are way out of their leaegues, and some are trading on reputations they either don't deserve or ceased to justify years before.  Here's a little story you'll find amusing.  Several years ago, I discovered that a fairly common complication of an operation done frequently around the world had been misdiagnosed historically and was being treated incorrectly for decades. What was thought to be a mechanical distortion of a specific cartilage that has to be cut during the procedure turned out to be a benign cartilaginous tumor triggered by the trauma of surgery.  When reoperating the second such case in my own patients, I sent the tissue to our pathology department - and the surgical pathologist with whom I worked most closely (who was also Chief of Surgical Pathology) confirmed with several specialized tests that it was a traumatic chondroma, as I suspected.

 

I then gathered info from colleagues around the country on similar cases and learned that simple reshaping was doomed to failure.  So I did an IRB supervised prospective review of every such patient I treated, removing the offending cartilage for pathologic examination and reconstructing the area with cartilage from the ear.  When we had 9 cases with identical pathologic findings, we wrote it up and submitted it to one of the most prestigious journals in my field.  It was rejected on the basis of this reviewer's comment: "I do not see any evidence of the pathologic process postulated by the authors.  I suggest that they have their specimens reviewd by a competent pathologist".

 

Once the steam stopped coming out of our ears, I sent the manuscript and all the slides to the gentleman who was senior head and neck pathologist for the Armed Forces Institute of Pathology at the time - his nickname for decades was "dean of American head and neck pathologists", and he was in fact that good.  His response was sent directly to the editor of the journal with a copy to me - it read, "I do not understand why this was at all necessary.  I agree completely with the authors and suggest immediate publication".  And that's what happened.

 

I don't know who the blowhard was who recommended rejection, and it's probably better if I never do.  He or she was clearly another surgeon with an axe to grind but little or no pathologic expertise (and may even have done the wrong operation for this so many times that there was concern about litigation if this was published).  So I have no problem identifying with the righteous indignation that should follow a flawed peer review.  But the process is a well established and long proven approach that produces far more good than bad results.

[Solstice380]

My experience is the same as @bluesman ‘s when I reviewed for Physica B, JAmCeram and and the ASM Journal, and those were all hard science, materials related journals.   

[The Computer Audiophile]

Great information on how these things work. 

 

I still don't like it. Putting one's name on something is the ultimate in transparency and honesty. In fact, I see some people advocating for use of real names on the internet because anonymous people behind keyboards can be a bad thing. 

[Solstice380]

24 minutes ago, The Computer Audiophile said:

Great information on how these things work. 

 

I still don't like it. Putting one's name on something is the ultimate in transparency and honesty. In fact, I see some people advocating for use of real names on the internet because anonymous people behind keyboards can be a bad thing. 

Then one of the authors of a cold fusion paper comes and shoots you for calling out their shoddy work.  No thanks!  (although those reviewers really missed that one!)

[The Computer Audiophile]

Just now, Solstice380 said:

Then one of the authors of a cold fusion paper comes and shoots you for calling out their shoddy work.  No thanks!  (although those reviewers really missed that one!)

You'll never stop crazy :~)

[Solstice380]

Just now, The Computer Audiophile said:

You'll never stop crazy :~)

2020 proved that without a doubt!

[R1200CL]

Do you have to pay to be an AES member ? Annual fee ?

Thinking of joining Chris ?

[yakman]

On 1/5/2021 at 2:27 AM, botrytis said:

  I was also told, that my reviews were too harsh because they were friends of my mentor and he just wanted them passed through. I would not do that and was put in the dog house due to that ( very apropos for UGA - 'Mericans will get the pun there).

 

Sorry to say that any publication with peer review process requires reviewers to declare "conflict of interests". Your mentor had definitely broken this rule..

 

I had rejected peer review invitations simply because I know the authors or they are my friends/mentor's friends.

Some time this can be hard to avoid as certain field has very limited researchers and most of them do know each other.

I don't think audio has gone that yet. Certain niche topic in audio may though.

 

Back to AES,

I have reviewed many research papers including those from AES publications.

I have to correct the OP, many recent AES convention papers had gone through peer review process. However the process for convention paper are more relaxed compared to journal. 

 

You can question whether AES publications represent the state of art of audio in every aspects. But calling AES publications not scientific is probably a bit too much.

Btw, i'm not a current AES member or holding any position in AES organization.

[opus101]

4 hours ago, R1200CL said:

Do you have to pay to be an AES member ? Annual fee ?

 

Its currently $125 for a single year, discounts are available if you wish to pay in advance for future years.

 

(I'm an ex-AES member).

[bluesman]

9 hours ago, yakman said:

However the process for convention paper are more relaxed compared to journal.

If that's now the case, it's changed since I was a member.  The peer review document used throughout my years as a member (linked in my post above and again HERE) clearly says that the review process for full papers submitted for pre-publication presentation at a conference is "...as a rule, the same as for a direct submission to JAES".  Exceptions are only made for a few unusual meetings, e.g. "new and emerging topics", for which peer review is left to the organizing committee for the conference.

 

Abstracts and precis that are submitted for presentation and subsequently written up as formal papers are also given full peer review before publication.  I don't recall that JAES had any obligatory publications when I was a member.

 

Of course, this could have changed - but I can't find any documentation of change if it did occur, and the standards for scientific journal publication have not been relaxed anywhere else.  They've been tightened.

[yakman]

5 hours ago, bluesman said:

If that's now the case, it's changed since I was a member.  The peer review document used throughout my years as a member (linked in my post above and again HERE) clearly says that the review process for full papers submitted for pre-publication presentation at a conference is "...as a rule, the same as for a direct submission to JAES".  Exceptions are only made for a few unusual meetings, e.g. "new and emerging topics", for which peer review is left to the organizing committee for the conference.

 

 

The guideline has not changed. But think of it, for convention papers, reviewers are typically assigned multiple papers and expected to feedback in a week or two. For Journal papers,  only one paper and reviewer has a few weeks to spend. All the reviewers are doing this as a volunteering service, many outside their daily job. How is it possible not to relax the rule when you're reviewing the convention paper?

This doesn't mean AES convention papers represent poor quality though.

For those outside the academic/scientific field, one paper typically get more than three reviewers.

[andrewinukm]

Accusing before seeking to understand the process is a common flaw. I am in full support of information transparency.

 

However, there are many considerations when deciding what level of transparency. Information transparency does not guarantee non-bias, it only means more information is available. More information could be a good thing, or it could make things worse. 

 

The reviewers didn't even write the papers, and here we are, arguing if the reviewers names should be published. Then where do we stop? Should they publish the names of lecturers who marked the reviewers' test papers in college? 

 

If a person is a subject matter expert, the reviewers will matter little. This person would be able to assess the quality and validity of the paper using his expertise. For the layman, any form of information within the paper is prone to be misinterpreted anyway. Declaring the reviewers only means one more thing to be misinterpreted, while it does nothing to reduce bias in the scientific publication process.

 

 

[The Computer Audiophile]

16 minutes ago, andrewinukm said:

Accusing before seeking to understand the process is a common flaw. I am in full support of information transparency.

 

However, there are many considerations when deciding what level of transparency. Information transparency does not guarantee non-bias, it only means more information is available. More information could be a good thing, or it could make things worse. 

 

The reviewers didn't even write the papers, and here we are, arguing if the reviewers names should be published. Then where do we stop? Should they publish the names of lecturers who marked the reviewers' test papers in college? 

 

If a person is a subject matter expert, the reviewers will matter little. This person would be able to assess the quality and validity of the paper using his expertise. For the layman, any form of information within the paper is prone to be misinterpreted anyway. Declaring the reviewers only means one more thing to be misinterpreted, while it does nothing to reduce bias in the scientific publication process.

 

 

Reviewers give their stamp of approval, which equates to a very powerful message for those who aren’t learned in a subject. Because peer reviewed papers are often used as badges of honor toward consumers, we should know who stamped it. If the process was so sound, it should be even better for each paper if we know that touted engineer Joe Public approves it. On the other hand, if the only people available to review a paper were far less than experts, but a paper needed to get published, so they went through with it, people should know. 
 

Transparency builds trust. Opacity builds skepticism. 
 

In my view anonymity is the old school minister of information style. The trust us because we know better style doesn’t build confidence. 

[Kal Rubinson]

1 hour ago, The Computer Audiophile said:

Reviewers give their stamp of approval, which equates to a very powerful message for those who aren’t learned in a subje

In a scientific journal, the role of the reviewer is to insure that the work was done properly but there is no stamp of approval on the results or the conclusions.

 

1 hour ago, The Computer Audiophile said:

Because peer reviewed papers are often used as badges of honor toward consumers, we should know who stamped it.

The audience for peer-reviewed papers is not the consumer.  It is the relevant scientific community.

 

2 hours ago, andrewinukm said:

For the layman, any form of information within the paper is prone to be misinterpreted anyway.

Very likely.  

[The Computer Audiophile]

5 minutes ago, Kal Rubinson said:

In a scientific journal, the role of the reviewer is to insure that the work was done properly but there is no stamp of approval on the results or the conclusions.

 

The audience for peer-reviewed papers is not the consumer.  It is the relevant scientific community.

 

Very likely.  

I certainly hear you Kal, but I can’t count how many times I’ve received a marketing pitch that included an AES paper mention to bolster credibility. 
 

If all of this stayed in the domain of the scientific community, it would be fine. It’s the crossover for marketing that causes issues. 

[Kal Rubinson]

28 minutes ago, The Computer Audiophile said:

I certainly hear you Kal, but I can’t count how many times I’ve received a marketing pitch that included an AES paper mention to bolster credibility. 
 

If all of this stayed in the domain of the scientific community, it would be fine. It’s the crossover for marketing that causes issues. 

Right.  The onus is on the marketing abuse on not on the source.  Hardy uncommon in many fields.

[botrytis]

17 minutes ago, Kal Rubinson said:

Right.  The onus is on the marketing abuse on not on the source.  Hardy uncommon in many fields.

 

The problem is the source is allowing the marketing abuse. AN example is the paper that was published about vaccines and Autism, that vaccines cause it. The paper was finally pulled and disavowed after a through and lengthy research into the author and his methodologies. The author was also convicted and jailed for his malicious lies and mistruths.

 

The problem is, once the genie is out of the bottle, it is hard to get back in. Anti-vaxxers still quote this paper as truth, even though it has been proven as a sham and a lie. Once something is said, on the internet, it is there forever.

[bluesman]

5 hours ago, The Computer Audiophile said:

Reviewers give their stamp of approval, which equates to a very powerful message for those who aren’t learned in a subject. Because peer reviewed papers are often used as badges of honor toward consumers, we should know who stamped it.

That's not the way it works, Chris.  As Kal points out, the purpose of peer review is not to validate the conclusions drawn by the authors - it's to validate the  study's premise, design, methodology, clarity, scientific soundness, originality, and educational value to the intended readership(s) and community.  A good reviewer knows the subject and its literature well enough to evaluate each submission for these and other qualities. I've reviewed over 1000 submissions to the top journals in my fields since 1979 (when I finished my residency).  My recommended rejection rate (a stat kept by the best journal editors for each of their reviewers) has run between 75% and 95% over 40 years, which is typical for editorial board members and reviewers at these journals.  But more important is the fact that even though I disagreed with the conclusions of at least half of the authors whose works I reviewed, I recomended their publication because they were well done, well written, and reasonable - and the main editor agreed with me on each.  The existence of multiple well done, peer reviewed studies that seem to contradict each other is a strong indication for more research.

 

A scientific study is only publishable if it adds useful information to the existing body of knowledge, i.e. if it identifies a new or novel finding, if it confirms either a prior study done on a scale too small to justify generaliztion or a study whose reported findings and interpretation go against conventional or curently accepted thought, it raises reasonable doubts about previously accepted findings, etc.  A good scientist is always open to new data and sound interpretation that contradict his or her findings and beliefs.  We can all be wrong and welcome sufficient input to clarify, supplement, or correct what we think we know.

 

Top quality peer reviewed scientific publications are not intended to support consumer or industry activities - they're only intended to ask a question and help answer it.  Any study with the occult but intended purpose of influencing market behavior in a given direction is biased beyond objectivity and is both unethical and grounds for academic discipline.  Every industry works with academic partners to study and develop new things - but every study that's done at any reputable institution must have sufficient scientific merit and importance to merit peer reviewed publication. 

 

Industry does sponsor a lot of reasearch.  But reputable researchers submit their work for publication, and meritorious submissions are published by top journals, regardless of the value of the information to the sponsor.  One of my departmental colleagues and his team were sued by a pharmaceutical company who sponsored a study in which we learned that waiting 12 weeks was as effective in resolving the problem of interest as was taking the sponsor's new drug for 12 weeks. The sponsor actually sued him to prevent publication - and they lost.

 

I've held professorial rank at the two largest and best universities in our region with full complements of educational and clinical programs in healthcare  (Assistant Prof 1979-82, Associate Prof 1982-93, Professor from 1993 to the present).  Several of my colleagues have been disciplined, demoted, and even dismissed for published improprieties.  Unfortunately, we can only control this from the supply side.  Once a study is published and someone gains legal access to it, there's no legal way to prevent its use for profit or other personal aggrandizement short of provable theft if IP, patent violation, etc.

 

We (i.e the academic and scientific community) have historically tried to prevent public dissemination of a lot of basic research for exactly this reason. It was not until recently that you could even subscribe to or otherwise access top peer reviewed medical and scientific journals without a credentialed subscription or through credentialed membership in libraries with institutional subscriptions.  For a lot of reasons (most purely economic - the scientific publishing business has been grossly altered by the internet), you can now buy subscriptions to good journals and even buy individual articles from many.  It's a tug of war for sure - but changing the peer review process offers little or no hope of improving anything I can identify.

[Bill Brown]

Kal and bluesman are speaking very, very wisely on this topic.....

 

In medical research and publication there are certainly controversies (especially re. the publication rate of negative studies as bluesman described above), but the identities of the reviewers isn't one of them.

 

Bill

[Kal Rubinson]

4 hours ago, botrytis said:

The problem is the source is allowing the marketing abuse. AN example is the paper that was published about vaccines and Autism, that vaccines cause it. The paper was finally pulled and disavowed after a through and lengthy research into the author and his methodologies. The author was also convicted and jailed for his malicious lies and mistruths.

Clearly, their review process failed.

4 hours ago, botrytis said:

The problem is, once the genie is out of the bottle, it is hard to get back in. Anti-vaxxers still quote this paper as truth, even though it has been proven as a sham and a lie. Once something is said, on the internet, it is there forever.

That is unfortunate.  Would public revelation (and/or shaming) of the reviewers have made any difference?

[botrytis]

9 minutes ago, Kal Rubinson said:

Clearly, their review process failed.

That is unfortunate.  Would public revelation (and/or shaming) of the reviewers have made any difference?

 

It might have. People did comment that the paper was wrong, but the author basically assailed anyone who questioned his article. Maybe if people would have looked closer and questioned more, that article might have never came out.

 

As far as MQA is concerned and after seeing the RMAF video of what happened to Chris, one phrase comes to mind, 'Thou dost protest too much.'

[Confused]

12 minutes ago, Kal Rubinson said:

Would public revelation (and/or shaming) of the reviewers have made any difference?

Arguably, the reviewers might have been more careful, or less blasé you might say, if they had known from the outset that their names would be published with the paper.  

[Kal Rubinson]

58 minutes ago, Confused said:

Arguably, the reviewers might have been more careful, or less blasé you might say, if they had known from the outset that their names would be published with the paper.  

Arguably, the editor might have selected better reviewers.  The point is that the error here is at the source.