Audio Blind Testing

[fas42]

The biggest problem in audio at the moment is, "Testing equipment issues". The Foobar200 ABX mechanism is hopelessly flawed, because its internal processing changes what is being tested, and degrades the quality substantially; Lacinato ABX has the potential to be far superior, but I haven't assessed it either.

All the other things mentioned are irrelevant if the core functionality is not right - good for debating sessions, only ...

[fas42]

20 hours ago, GUTB said:

I have nothing to add to the topic of audio blind testing -- except to say that it won't work unless there are very large differences in sound. Not to any degree of acceptable mathematical rigor, anyway. This is due to the ear-brain issues that are difficult to control for.

 

I attempted to take the concept of ABX seriously, some years ago, and determined that the tools available were so poor, or were unavailable - so, lost interest fast ... . Specifically, Foobar's extension has so many issues that it was a waste of time - a badly made hammer is useless, as a tool.

 

In other areas where hardware is involved the difficulties are horrendous, especially in the areas where I operate - altering part of the system where, say, hard wiring is an essential means that the exercise is an impossibility.

 

The point, yet again, is to completely forget about whether something is different - this is useless as a means of advancing the status, performance of an audio system; a complete waste of time. The only criterion should be whether one can detect whether the playback is audibly faulty or not - and then resolve any failings ... if a luxury car has a rattle in it, then the only concern is to get rid of the rattle - not, "do you prefer this quality, or that quality, of annoying noise?"; or, "if you increase the thickness of the carpet it makes it harder to hear the rattle!" .. I shake my head a lot of the time when I read comments about audio ...

[fas42]

41 minutes ago, GUTB said:

 

A very valid approach. You can look at your system as a constant journey to fix problems. In my case, the big problem to be addressed is lack of dynamic force. Other problems are that my soundstage isn’t fully unfurling and little bass response (which is admittedly done on purpose to avoid destructive room modes).

 

That's the idea!

 

Immediately one can consider what needs to be addressed, having stated those concerns - as an example, a lack of dynamic force implies that distortion levels are building too fast as the volume rises, which could be due to a number of factors, each of which can be looked at in turn. Personally, I would now consider whether the speaker are sufficiently stabilised in their location, whether the power supplies of the amplifier were sufficiently sorted, and how well the components were isolated from each other's impact on power supply noise - as a couple of starters.

[fas42]

39 minutes ago, knickerhawk said:

To start the “randomization” I just randomly scrolled through the playing queue at the bottom to approximately the midway point in the queue and selected one of the identical thumbnails. As long as I avoided scrolling to a starting point that showed either the first or last item in the queue it was impossible to tell which version had been selected. After listening to the first randomly selected track, I then listened to it’s “neighbor” (which because of the a/b sequencing of the queue I knew was from the other format). I went back and forth several times until I was ready to pull the Post-It off and see which track was which format. Worked like a charm for allowing me to do a “blinded” personal a/b comparison of CD and MQA versions in Tidal!

 

Good one! I've had ideas of doing a variation of this, to satisfy those who just have to have "somethin' scientific" before they believe anything. There's clearly audible differences between MQA and without - the point then is whether MQA is just "distorting" the raw version to make it "nicer" for a high percentage of people.

[fas42]

15 minutes ago, gmgraves said:

 

Actually, I came to the conclusion that these interconnect cable differences were imaginary after they disappeared in DBT after DBT. It was then that I realized what a strong influence expectational and conformational bias plays on the human brain. Senses are easily corruptible and can only be trusted to a certain degree. The truth is that people see and hear what they expect to see and hear and often what they want (consciously or subconsciously) to see and hear. Criminal science has decided that no evidence in a court of law is more unreliable than the well-meaning eye-witness. Unfortunately, the court systems (at least in the USA) have yet to evolve to the point where eye-witness testimony is accorded the diminished weight it deserves in an actual trial. 

 

Things like interconnect cable differences will always fail in DBT - that's because the time factor aspect is never part of the test - first rule of science: always make sure you are really testing what you think you are attempting to verify, rather than doing a show off demo of the fact that you can play "scientist in a lab coat".

 

Cabling behaviour varies over time - because of material considerations - there are all sorts of subtleties involved here, which matter when the best in reproduction is being aimed for. Claiming that this sort of thing doesn't exist, "because it shouldn't !! ", is not very helpful ...

[fas42]

30 minutes ago, Ralf11 said:

claiming it DOES exist with no evidence is a LOT worse

 

 

 

Everyone accepts that speakers sound different, because different materials are used in the design and assembly of the drivers - no-one gets excited when someone claims that speaker A sounds different from speaker B. Yet, this all evaporates when we move from the nominally mechanical world of speakers, to the nominally electrical world of the other components - so, do we believe that electrical behaviour is a magical aspect of nature that behaves precisely as the textbooks state, by very simple rules, under all circumstances?

 

I don't - the world of audio is beset with parts that have all sorts of parasitic behaviours - they don't behave just like the textbook says - because, hey, they are made by humans, using imperfect manufacturing processes - I would be immensely surprised if some electrical part was "perfect" - which includes cable. The more I've gone into it, the messier it gets - a good system is always a balancing act of compromises: get right what is most important; be less fussy with the rest.

 

Claiming that some part of an audio system couldn't be better in how it functioned would be a true absurdity, IMO. The argument is really whether it always does its job well enough so that it never is audible, in any situation - my experience is that the better a system gets, the more a nuisance the residual behaviours become - because their impact is no longer hidden under the "noise" of the more obvious shortcomings of the rig.

 

Now, if someone claimed that a luxury vehicle couldn't develop an annoying rattle, under any circumstances - then I would be extemely skeptical ...

[fas42]

25 minutes ago, sandyk said:

 The time factor is very relevant due to the length of time between cable changeovers.

 The problem here, is that even  if you switch using a relay type comparator, you are introducing other variables such as plugs and sockets and additional length compared with using just the cables themselves. 

 

I'm thinking in terms of it taking time for the cable materials, to settle down, stabilise after they've been manhandled (womenhandled?), and the metal to metal contacts in the path to be in a long term stable state - I hate this sort of thing, so use cheap, every day wire which I hardwire into the setup - and then leave it ... problem solved ...

[fas42]

32 minutes ago, pkane2001 said:

 

 

What does time have to do with the blind aspect of the test? If you think that a blind test must only be done with a rapid A/B switching, then you might be right, but who said you have to do that?

 

If you can tell a difference in a fully sighted test, then try to repeat exactly the same test, blind. Same time frames, same comparisons,  same material, except that you don't know which cable is in the system. If you fail to distinguish them while blind, you can be fairly confident that whatever differences you heard while testing sighted was not real. And if you're afraid that the pressure of a blind test or performance anxiety will skew the test -- take a Zanax.

 

 

 

As a following post emphasised, it's to do with all the materials in the path stabilising - as the most straightforward example, the metal to metal contacts at either end of the cable are initially "clean", from the wiping of the contact surfaces - then they steadily build up corrosion contaminants, which affects the sound. Slowly the construction of the cable comes into play, altering the spectrum of distortion artifacts.

 

Not something I've done myself, but an audio friend had normal, and pricey, audiophile cable. Clear difference between the two, used normally, he said; then he took on my suggestion to hardwire the links in the system ... and the differences between the everyday, and "special" cable went away! That convinced him ... I am not interested in listening for different types of distortion, I want no distortion - hence, create air tight connections everywhere that matters.

[fas42]

14 minutes ago, Ralf11 said:

One good way to do listening comparisons is listen to one short passage on A, and then on B

- this is trivial to setup using 2 different CD players; not so easy with 2 different speakers

 

*** ...not to rule out extended sessions

 

An easy method for picking audible variations in versions of source files, that I have used on occasion, is to bring both into an editing program like Audacity, make sure they are fairly closely synchronised - and pick a likely area that may vary, say a few seconds worth. Set up a loop of playing that spot, continuously, and solo, select one track only - let it build up a rhythm of sound in your mind, almost like a mediatative thing - and then, switch the track being played in the loop. Often times, the difference will hit you like a solid thump - or, it may not vary a beat in the sense of it ... you have an answer, just like that.

[fas42]

1 hour ago, pkane2001 said:

 

Regardless of any supposed 'settling' or 'break-in' issues related to cables, the point is that a sighted test is subject to exactly the same constraints as a blind test as far as time is concerned. What would cause a cable longer to stabilize in a blind test compared to the same cable in a sighted test?

 

Out of curiosity, do you have any references to measurements or other objective studies of  cable stabilization time?

 

We could be talking hours, days for conditions to stabilise - makes it difficult to run ABX under those circumstances! Other times, the variation is almost immediate - depending upon precisely what is is causing a 'problem'.

 

Things taking a long time to stabilise are a curse in any field, I used to run my system 24/7 decades ago, because of this behaviour - a waste of power, etc, but I hated the loss of quality that occurred every time there was a switch on from cold, which took ages to settle down.

 

I can't recall coming across any measurement data, or other studies - usually anecdotal, which would draw an "Ah Hah!" of recognition from me ... the best I can do here.

 

As I've said a number of times, my goal would be for a system, assembled from scratch, to reach acceptable quality about 5 minutes or so after power on - I've never achieved this ... perhaps down the track ...

[fas42]

34 minutes ago, pkane2001 said:

 

Frank, that's a very tenuous explanation for why a DBT might fail to detect a difference. You are basing this on anecdotal evidence of cable 'stabilization'. And, you are missing the point, again: the same cable behavior applies whether or not you are doing sighted or unsighted testing. So, any findings of different sound between cables (or no difference) is just as valid or invalid in both cases.

 

There's nothing special that makes a DBT test more susceptible  to timing due to cable stabilization than a sighted test. Unless, of course, you claim some quantum mechanical effect related to a wave function collapse in a sighted test

 

 

 

Hey, I don't want to get into a full blown argument of why conventional DBT may or may not detect differences, re. cables, . If one wanted to pursue it as a more rigorous exercise the only way I could see it capable of giving worthwhile results would be to have two identical systems, carefully assessed to be effectively impossible to distinguish; and then use cable A on one, and cable B on the other - this would take stabilisation out of the equation.

 

Cables are boring ... so long as they don't affect the sound is all I worry about - and basic stuff is fine, so long as they are installed well.

[fas42]

6 hours ago, marce said:

Provide some evidence of what you claim, we can easily measure and show difference is cables that are so low to be inaudible, all cables are different to an extent. It is whether this difference is audible, and mostly it isn't, unless the cable is either bad or designed to sound different!

 

Your claim is that differences in cable materials, construction, how it is connected into the circuit is always inaudible - many who have investigated this as an experimental exercise, as compared with doing a quick switcheroo to confirm a prejudged notion, have found otherwise. I would never have got the quality of sound I talk about, and still couldn't; if I didn't take this aspect seriously.

 

Many audio people can never get the concept, no matter how many times it's repeated to them - a chap with a Ferrari wants to know which are the optimum tyres for him driving his car close to its limits; someone who has a shopping runabout says, he's tried the very expensive, top technical rating variety, and it made not an ounce of difference to the driving experience - well, that was good advice, wasn't it ... ?

 

Which doesn't mean buying expensive stuff - it means, that if one wants more than average performance then everything needs to constantly assessed, to make sure it doesn't make a difference ..

[fas42]

Sound levels don't matter, one iota - if one knows what to listen for  - as soon as the word "better" is tossed around in a conversation about audible differences, I roll my eyes - it means that the people haven't got a clue about how to assess SQ, and perhaps will never have ...

[fas42]

1 hour ago, sandyk said:

 

For starters, nobody in their right mind would use a Zip cord for an Interconnect !!!

 

 

Ummm, bits of wire lying around is good enough, most times - if used well. Current experimental rig, NAD components, uses a bit of a single twisted pair, solid strand Cat cable between source and amplifier - works well enough to make all the other weaknesses which are holding back the sound obvious. Perhaps it will be worth making that connection "better" down the track, perhaps not ...

[fas42]

7 hours ago, audiventory said:

 

Comparison of high resolution, SACD and CD is technically impossible. Even for single DAC. Because different electrical circuits work for each of these modes (see "PCM versus DSD DAC" part): https://samplerateconverter.com/educational/what-dsd

 

And this is the point. Playback quality is a result of the full chain in operation, source to speakers. Thinking otherwise is as silly as listening to a demo at an audio show, which "proves" how good an amplifier is - ummm, you mean the player, DAC, preamp, speakers, room, setup skills have absolutely nothing to do with it ... ?

[fas42]

56 minutes ago, sandyk said:

 

  But you live in an area (lower Blue Mountains ?) which is well removed from the industrial areas of Sydney, and the vast majority of high powered AM, FM Stereo, DTV and Radar transmitters.(Weather and Aviation etc.)

There is a big difference between a Zip cable and a twisted pair which has been used since the evolution of Telecommunications to improve signal integrity.

 

Yes, the point is signal integrity. If that is focused on, and made sufficent for the situation, then no more has to be done - I have also used basic electrical cable for speaker links, but, no stranding, and tightly twisted - it's all about construction, and how the 'part' is used ...

[fas42]

51 minutes ago, gmgraves said:

With active components, it's different. There are so many variables, and it's so easy to get a mismatch in level between two DUTs (even 1 dB can be enough to invalidate the test) that in my opinion, one is better off skipping the DBT altogether and going with long term comparisons while taking lots of notes! 

In the post to which you reference, I stated the fact that in a number of different DBT "cable shoot-outs" to which I've been party, Instantaneous switching between cables has concluded in a result where not only has no one participating been able to hear any discernible difference, they couldn't even tell when or if the cables had been switched. That's my experience, and I'm content that interconnect cables aren't worth my time. Others might see it differently, that's up to them. But in the context of this thread, I feel that my comments were on subject and germane to the topic of DBTs. And no, I don't wish to visit the interconnect question as a subject for debate again. To me it's a question asked and answered.

 

Ah, dear ... you've answered your own "questions" - it's the "long term" consideration aspect, you see ...

 

Which is the only way to make decisions ... there's a concept which is an inconvenient truth in audio, being, that qualities change over time, for a myriad of reasons. And no matter how hard one wants to pretend that a plugged in cable, as a piece of an electrical circuit puzzle, is 100% passive - whatever that means - the reality is otherwise. A genuine, engineering solution is to make all the electricals fit in one box - no more cable nonsense! - but then we wouldn't have the fun of trying a Chrysler engine in a Ford body, with a GM suspension - surely that's gotta be better than a car done by a single engineering team ... .

[fas42]

6 hours ago, gmgraves said:

 

No, my results have answered my questions, not I. And if things change over time, then they change over time, but if you can't notice or measure the changes, what difference does it make?  I mean it's a lot like the classic dilemma; "If a tree falls in the forest, and there is nobody (and nothing) around hear it fall, does it make any sound?" The answer, is of course, that it doesn't matter and neither does this.  

 

Things changing over time are very, very important to me ... because the first time I experienced a step jump in subjective quality of playback, demonstrating what was possible, a huge frustration was that this major difference in the subjective presentation was very fragile - it would vanish like the morning mist, as I was listening - over and over again. I could not get an upper hand on this behaviour at the time, but it made me enormously aware that the slightest "imperfection" in the setup could be enough to degrade key areas of integrity - which I've spent years tackling, on and off.

 

It is entirely possible that a system may be so capable in every other way that a non-optimised interconnect won't be audibly significant - but I haven't come across such a situation yet. When a majority of systems can produce competent sound, without taking special care, then the relative importance of such things will be easier to determine.

[fas42]

4 hours ago, audiventory said:

 

I meant that comparison of 2 resolutions or DSD vs. PCM is technically non-correct (methodologically impossible) in general case. It do not depend on equipment.

 

I was agreeing with you, but made a poor attempt at humour, regarding how people can "know" what is the cause of something being better or worse in a complex system, when there is almost no information ... sorry for the confusion!

[fas42]

2 hours ago, gmgraves said:

 

If I understand correctly what you are saying, I see people making this claim all the time, and frankly as a many year owner of tube gear, I don't buy it.  People cannot detect gradual changes in an audio system's sound (an analogy is the frog in the pot of heating water). Many's the time I have been astounded at the difference when changing-out older tubes in an audio system. I never noticed the gradual deterioration of SQ until I changed tubes, and then the difference was considerable and immediately noticeable. Also, human beings have no long-term memory for sound quality either. So when I hear people talk about how their components "burn-in" over many weeks or months, I look upon that statement with jaundiced eye, because nobody can possibly remember what something sounded like weeks or months before. If one insists that they can, then I say that person is either delusional or a liar. 

 

 

For me, the change is a profound, subjective one; and has very distinct, rock solid characteristics. The most 'impressive' one is that the speaker drivers become completely non-locatable - if you had a blindfold on, you could stumble around the room and no matter where you were in relation to a particular speaker, you would be unable to point to a driver making the sounds. When I first got a system working this well that quality lasted about ten minutes, and then steadily vanished - it was back to normal audio, meaning it was obvious that the right, and left speakers were the source of the sound.

 

Which means that all I worry about is whether that 'switch' is on, or off - invisible speakers, or normal? The problem is, that everything in the entire system has to be working with a high level of integrity, and just one tiny thing not quite good enough is enough to kill this illusion.

 

Quote

Can you please explain what a "non-optimized interconnect", and how does one know that they have found an "optimized" one? There are thousands of interconnect brands and models out there. How does one know that the interconnect that they just bought is "optimized" over the thousands of other choices of interconnects that they could have bought?

 

The interconnect should have the electrical characteristics of a short, soldered link or track on a circuit board - anything less than that could cause audible issues. The technique I use is single core wire, shielded as best as possible, no longer than necessary, and soldered directly to the appropriate circuit nodes at either end - the connection has lost the properties of a plug-in cable, and everything benefits. Of course, this option is not open to most people - and this is something the audio industry needs to address.

 

If a plug-in cable has something about its construction that gets it closer to not causing audible degradation, in a particular system, then it will be better - and that is something that's mighty hard to predict.

 

[fas42]

8 hours ago, audiventory said:

It mean that all the cases are possible:

1) PCM better DSD

2) DSD better PCM

3) No difference

 

We can't claim that DSD better PCM (and contrary) for all cases.

 

Example #2:

 

1) 24 better 16 bit

2) 16 better 24 bit

3) No difference

 

We can't claim that 24 better 16 bit (and contrary) for all cases.

 

I don't buy into any of the hires, SACD, DSD thing - classic Redbook CD has always had everything for optimum sound. The differences in the output sound are because of the implementation of the playback circuitry, and if the latter is done well enough then format is completely irrelevant.

[fas42]

51 minutes ago, gmgraves said:

I seriously doubt that you are a liar. All psychoacousticians agree that human memory of how something in a musical performance sounds is extremely short (strangely though, this does not apply to voices. We recognize voices that are familiar to us almost instantly and we never forget them. This is tied to some primordial survival skill). Oh, you can concentrate on one aspect of the sound and remember it as a general impression, but not in fact. What I mean by that is something like: "I went to the symphony last night and I couldn't get over how smooth the strings sounded." Now you will remember your impression of the strings, but you won't be able to remember what the strings actually sounded like, just your reaction to what they sounded like. In your case you have a mental impression of some aspect of past imaging, and you are comparing that to a current impression of imaging, but here's the rub. Your remembered impression of anything really specific, might be inaccurate. Specific impressions are very subject to the vagaries of human memory. An excellent example of this is the eye witness to a crime who is sure that he saw the defendant commit the crime, and is later found out to have fingered the wrong man. A lot of research has been done in this field recently. It turns out that memory in humans doesn't work like a recording (which was supposed for decades), that is to say, it's not continuous. The brain "refreshes" the memory every time it is brought to consciousness. IOW, it recalls it and refiles or 're-writes" it. When this happens, all kinds of non associative forces creep into it changing it subtly. Not saying that this is happening in your example, but it's possible simply because you are human. 

 

 

Which is why I have not the slightest interest in what a "system sounds like" - this is irrelevant. What matters is whether elements of recordings that I know well are reproduced without obvious problems, and virtually all systems I casually come across fail by this test - it's defective behaviour of the playback chain that's audible, that needs to be addressed; which automatically leads to optimum reproduction of all one's recordings.

[fas42]

4 minutes ago, gmgraves said:

If you are saying that RCA connectors suck, you won't get any argument from me. How this connector made the jump from a connector invented by RCA Victor in the mid-forties to transfer the IF signal from the tuner of an RCA model 4630 TV (the first commercial post WWII TV set) to the first IF amplifier on the main chassis, to the ubiquitous universal Hi-Fi interconnect connector, is beyond me!

 

 

It's a huge weak link ... the only solution is to completely bypass it; the first thing I do when optimising a setup. Otherwise, I might as well try driving a Ferrari with flat tyres ...

[fas42]

19 hours ago, sandyk said:

Much of the testing I do after modifications involves listening to material where there is an excellent 3D type image.

Either the illusion sounds very real , or it is missing something compared with how well you know that it SHOULD sound. 

 Occasionally, I will be surprised at how good it sounds this time, and realise that it is a further improvement.

That is very different to the situation you quoted.

 

 

Alex, wanted to comment on this yesterday, but didn't make it! The "3D quality" of the presentation is another way of describing attributes of higher level playback; and this happens, automatically, when the low level detail in the recording is clearly rendered - the ear/brain can now make sense of what "all that muffled stuff" means - and full clarity of the sound field is perceived.

 

Do you have to remember what a violin sounds like in the flesh, versus the conventional representation by audio systems? If you do, you have my sympathies ... most people have never experienced a rig that does things like getting instruments so right that it is impossible to pick the fake; comparing two dodgy imitations of such is rather pointless, IMO - the only metric worthy of taking seriously is how close to achieving that convincing standard one is.

[fas42]

1 hour ago, Ralf11 said:

Music lovers have no interest in DIY capacitor de-soldering either !!!

 

Your arrogance in attempting to make decisions for "Music lovers" is one of the things that earned you Audio Ayatollah status - a nom de diss I agree with.

 

Of course, this is the big problem, in the industry - Alex and I don't hesitate to make changes to equipment, to solve problems - and this often is the only way to resolve the underlying weakness. If a quality bottleneck exists, and can't be sorted external to the components, then those who don't have DIY skills or inclination are in a bind! A change in attitude, in the audio field overall, is needed - and hopefully this will come about down the track ...