Bits, bandwidth and information theory, was...

I am creating a new thread from this

So here's my layperson's take, and you can tell me where I'm wrong so I learn something (which for me is the exciting part):

 

Working with a Shannon-Kolmogorov definition of "information" (the intersection of Shannon entropy and Kolmogorov complexity, see, e.g., http://homepages.cwi.nl/~paulv/papers/info.pdf), gives us something very much akin to what we would think of as compressibility. Two files that can be losslessly compressed to the same size have equivalent amounts of information/entropy/complexity.

 

Take a RedBook file and do the interpolation and filtering required to convert to 24/192. Is there a mathematical operation that will convert the 24/192 file back to the RedBook file losslessly? (Does closed form filtering make any difference here?) If not, to me this necessarily implies that the 24/192 file cannot be compressed to the size of the RedBook file, and therefore under the Shannon-Kolmogorov definition(s), the 24/192 file has more information. Thus information was added by the sample rate conversion operations of interpolation and filtering.

 

Through all of this, I am agreeing with just about everyone who's talked about it thus far, including my own previous comments, that this additional information will do nothing to further specify the analog signal.

So the important issue is where we start and what we consider "lossless".

If you say that we are starting with a Redbook file (16/44) or with an analog audio signal which is entirely represented in this file (i.e. is dynamic range and bandwidth limited) then the upconversion to 24/192 does not add information. Any information in higher order harmonics, for example, was not present in the original signal and according to the Shannon (1948) framework, would be noise, and so according to Shannon, would be compressible back to Redbook. Remember that the entrance criteria of Nyquist means that the signal is appropriately bandwidth limited.

On the other hand if you start with a 24/192 file and this file contains higher order harmonics, then no, you can't losslessly compress down to 16/44. Shannon 1948 describes noise as being added in the transmission process, which consequently is removed, preserving the original signal. But if you start with a 24/192 file, and its not stated that higher order harmonics are noise, then you need to preserve these as "information"

Does that make more sense?

Sounds good.

Well, since Shannon/Nyquist is always a favorite topic, here's a little PDF I found a few years ago that saves typing in the starting math. Its a fun starting point!

Fourier2_2p.PDF

Consider this, suppose the intent is to transmit a covert signal in the ultrasonic band. Such a signal may be arranged to stochiastically simulate noise, so in general if you are given a 24/192 file you need to preserve it. But if you start with 16/44 there is no way to put information into higher order harmonics.

Well, since Shannon/Nyquist is always a favorite topic, here's a little PDF I found a few years ago that saves typing in the starting math. Its a fun starting point!

 

[ATTACH]24204[/ATTACH]

That's great! Here is a link to Shannon (1948) in case anyone wants to read the source: http://www.essrl.wustl.edu/~jao/itrg/shannon.pdf

Consider this, suppose the intent is to transmit a covert signal in the ultrasonic band. Such a signal may be arranged to stochiastically simulate noise, so in general if you are given a 24/192 file you need to preserve it. But if you start with 16/44 there is no way to put information into higher order harmonics.

In theory, one could embed low data rate information into the alias effects below 22.5khz. Make the sound only music to the ears of spies though...

If you have a 16/44 file, upsample to 24/192, the resulting file is fully defined by the original file AND the filter. applying an inverse filter can return it to the original 16/44. Thus the information content of the upsampled file is the original plus filter, which is a small addition but not a lot. BTW this assumes you know the filter, if you just have the upsampled file figuring out that filter is a non-trivial task.

So even though the additional information content is small, using a general purpose compression algorithm may not be very effective if it does not come close to the original upsampling filter.

John S.

That's a good explanation of the Kolmogorov approach

Not all information/entropy is equal. If I take a signal (e.g. a music recording) and add white noise, the result will by your definition contain more information since it can't be compressed as much. Does this mean the original+noise sounds better or is otherwise preferable to the original? Hardly. Any information added by a filter (interpolation or other type) is noise/distortion caused by finite precision implementations, i.e. something undesirable but unavoidable.

As previously noted, aside from the noise, "extra" data points unnecessary for reconstructing the original analog signal are added by interpolation. To the extent the file containing the interpolated data is not compressible to the size of the initial RedBook file, information has been added under the applicable scientific information theory definition(s). While this added information is unnecessary to reconstruct the original signal according to the Sampling Theorem, it does, as we've said (and as you say below) allow the reconstruction filtering to add back less noise of its own. So it is information, and it is useful; but it is not useful for the specific purpose of reconstructing the original signal.

A DAC consists, somewhat simplified, of the following stages (the first two optional):

 

1. Upsampling/interpolation

2. Sigma-delta modulation

3. D/A conversion

4. Analogue low-pass filter

 

Each of these stages introduces some amount of noise and distortion. The goal is to pick the parameters such that the total end-to-end performance is maximised (within cost/power/whatever constraints), and adding a little noise in one stage can let us reduce it a lot in another. For instance, the D/A stage is more accurate at a lower bit depth, enough so that it more than compensates for the (audio band) noise added by the sigma-delta modulation. Likewise, while a digital interpolation filter adds a little noise, it allows for an analogue filter with a higher cutoff frequency and thus less distortion in the audio band. The sigma-delta modulation also performs better the higher the sample rate.

 

When sound quality is improved by software upsampling or DSD conversion, it is because we are replacing the first stages of the DAC with a higher quality implementation (at tremendous (relative) expense in cost and power), not because it somehow adds information to the signal.

If the original redbook file can't be precisely reconstructed, information has been lost, not added. This loss is caused by various inaccuracies in practical implementations. A perfect (infinite precision, infinite time) interpolation is reversible. For an integer multiple interpolation, the reversal is trivial: drop all but every N samples.

Yes, thus my mention in the other thread of a closed form filter that retains the original samples.

I think you are being imprecise when you say if the initial RedBook file can't be precisely reconstructed, information "has been lost, not added." As we've seen, information has undoubtedly been added under the information theory definition(s). But you are correct that the information regarding the precise data points in the initial RedBook file has been lost (absent a closed form filter). Thus information has been lost and​ added.

That's a good explanation of the Kolmogorov approach

Jabbr,

Definitely off topic - is that John Yaya and John Bigboote with you?

jabbr,

 

definitely off topic - is that john yaya and john bigboote with you?

you can check your anatomy all you want, and even though there may be normal variation, when it comes right down to it, this far inside the head it all looks the same. No, no, no, don't tug on that. You never know what it might be attached to.

I'll take that as a yes.....

I am creating a new thread from this

 

so that we won't be off topic.

 

So the important issue is where we start and what we consider "lossless".

 

If you say that we are starting with a Redbook file (16/44) or with an analog audio signal which is entirely represented in this file (i.e. is dynamic range and bandwidth limited) then the upconversion to 24/192 does not add information. Any information in higher order harmonics, for example, was not present in the original signal and according to the Shannon (1948) framework, would be noise, and so according to Shannon, would be compressible back to Redbook. Remember that the entrance criteria of Nyquist means that the signal is appropriately bandwidth limited.

 

On the other hand if you start with a 24/192 file and this file contains higher order harmonics, then no, you can't losslessly compress down to 16/44. Shannon 1948 describes noise as being added in the transmission process, which consequently is removed, preserving the original signal. But if you start with a 24/192 file, and its not stated that higher order harmonics are noise, then you need to preserve these as "information"

 

Does that make more sense?

Everything you say is absolutely true if you view "information" as "that information required under the Sampling Theorem to reconstruct the sampled signal." So here we are in a digital audio forum where the Sampling Theorem is king. Why isn't viewing things in terms of the Sampling Theorem "good enough" for me? Why am I continuing to insist on use of the scientific information theory definition (which holds for Shannon entropy and Kolmogorov complexity alike)?

I feel that being imprecise and using a definition of "information" that is not used by scientists creates the danger of sneaking in a particular point of view under cover of apparent objectivity.

Here in the real world, we cannot perfectly reconstruct an analog signal, not because we have difficulty finding enough data points to accurately define the signal, but because we have neither perfect reconstruction filters nor infinite time to apply them. Keeping the analog signal at higher resolutions, or failing that, upsampling the RedBook bitstream to a higher resolution, allows use of filters that can do a better job of reconstruction because they add back less distortion in the process. Treating RedBook as if it's all that is necessary for best reconstruction is inaccurate, because it is only half the story. It only allows us to accurately define the analog signal, but not to reconstruct it best. We are also handicapped in explaining how DACs actually work to people less familiar with digital audio who think hi res is necessary to accurately define an analog signal, and/or that upsampled data points are somehow not "real," if we use imprecise definitions under which upsampled data points are useless, "not even information."

It's kind of a corollary to "If you don't lie, you don't have to remember what you said to whom." If you always use the scientific definition(s), you won't have to change your explanation depending on who you're talking to.

Jabbr,

 

Definitely off topic - is that John Yaya and John Bigboote with you?

Bigboo-tay​.

Everything you say is absolutely true if you view "information" as "that information required under the Sampling Theorem to reconstruct the sampled signal." So here we are in a digital audio forum where the Sampling Theorem is king. Why isn't viewing things in terms of the Sampling Theorem "good enough" for me? Why am I continuing to insist on use of the scientific information theory definition (which holds for Shannon entropy and Kolmogorov complexity alike)?

 

I feel that being imprecise and using a definition of "information" that is not used by scientists creates the danger of sneaking in a particular point of view under cover of apparent objectivity.

 

Here in the real world, we cannot perfectly reconstruct an analog signal, not because we have difficulty finding enough data points to accurately define the signal, but because we have neither perfect reconstruction filters nor infinite time to apply them. Keeping the analog signal at higher resolutions, or failing that, upsampling the RedBook bitstream to a higher resolution, allows use of filters that can do a better job of reconstruction because they add back less distortion in the process. Treating RedBook as if it's all that is necessary for best reconstruction is inaccurate, because it is only half the story. It only allows us to accurately define the analog signal, but not to reconstruct it best. We are also handicapped in explaining how DACs actually work to people less familiar with digital audio who think hi res is necessary to accurately define an analog signal, and/or that upsampled data points are somehow not "real," if we use imprecise definitions under which upsampled data points are useless, "not even information."

 

It's kind of a corollary to "If you don't lie, you don't have to remember what you said to whom." If you always use the scientific definition(s), you won't have to change your explanation depending on who you're talking to.

You've lost me here. I *am* using Shannon's and/or Kolmogorov's definitions, and I've referenced Shannon's seminal paper. This is the definition that is used by information scientists,

Do you have a different "scientific" definition in mind? Please reference this precise definition! (reference the paper and quote the definition that supports your point)

... and I've not once said that Redbook is sufficient, nor have I ever suggested that digital manipulation a like upsampling and filtering are not very practically useful in reconstructing the analog signal.

What I am saying is that in an idealized Shannon model the information present in the source analog signal is transmitted to the destination analog signal. Not more, not less, ideally unchanged. That would be Shannon's take home message.

 

What I am saying is that in an idealized Shannon model the information present in the source analog signal is transmitted to the destination analog signal. Not more, not less, ideally unchanged. That would be Shannon's take home message.

Yep, no disagreement.

What I'm then saying is that one can then take the digital bitstream that was sufficient to reconstruct the analog signal in the idealized Shannon model; and interpolate and filter it, and this interpolation and filtering creates additional information, so long as there's no mathematical operation that will perfectly reverse the result.

Two things can be said of this additional information:

- It is unnecessary for the purpose of accurately defining the analog signal.

- It allows the use of reconstruction filtering that adds back less noise/distortion in the process of reconstructing the analog signal, here in the non-idealized real world where there are no perfect filters.

Query if we were in the idealized world of the Sampling Theorem where perfect filtering is possible, would this negate the possibility of creating additional information through interpolation, since one could always, via perfect filtering, reverse the operation?

(Moved from the other thread:)

... But interpolation certainly can add new information, interpolating between the dat points to create new data that is meaningful information. Replacing information that was lost during the sampling operations. (*Not exactly, but close enough as to make no difference in any practical sense.) ...

I disagree. The information between the original samples is already there, contained within the samples. Upsampling adds more samples, but not more information. It especially does not replace information that was lost during the sampling operations. That was lost in the anti-aliasing filter and can never be retrieved. The best you can do by upsampling is generate samples that match what they would have been if you'd sampled at the higher rate but used the lower rate filter. Sampling at the "1x" rate already captures all the information the filter passed. Sampling at "2x" is 50% redundant. You can discard every second sample and still have all the information that the filter passed. Ditto for upsampling after the sampling.

It doesn't amaze me that people get confused by this. It's non-intuitive. If it was easy, we wouldn't need basic fact education such as Monty's "Digital media primer" and "Digital show and tell" videos.

I do agree that in the real world upsampling has its uses, especially for reconstruction filtering at the DAC stage.

Yep, no disagreement.

 

What I'm then saying is that one can then take the digital bitstream that was sufficient to reconstruct the analog signal in the idealized Shannon model; and interpolate and filter it, and this interpolation and filtering creates additional information, so long as there's no mathematical operation that will perfectly reverse the result.

I know that really good definitions are important to you, so let's quibble here about how you are using the term "additional information":)

According to Shannon (1948) this isn't "additional information" rather part of the transmission process. If the result isn't reversible, this is considered an error -- it is important for Shannon that the messages are transmitted error free.

Now according to Kolmogorov, there is a small amount of additional information, namely the algorithm used to interpolate/filter the signal, but nor does Kolmogorov consider the additional interpolated data points themselves to be additional information. This is an important point to understand. In neither of these information theories is there a 1:1 correspondence between data point and information -- explicitly not in fact.

Two things can be said of this additional information:

 

- It is unnecessary for the purpose of accurately defining the analog signal.

If not important for accurate definition of the analog signal then not "information", and can be optimized/compressed away.

- It allows the use of reconstruction filtering that adds back less noise/distortion in the process of reconstructing the analog signal, here in the non-idealized real world where there are no perfect filters.

That is an implementation detail, information only in so far as the reconstruction algorithm can be considered information.

Query if we were in the idealized world of the Sampling Theorem where perfect filtering is possible, would this negate the possibility of creating additional information through interpolation, since one could always, via perfect filtering, reverse the operation?

Well yes for Miska, there are perfect filters:) but nonetheless, the information theories do not consider imperfect algorithms as "adding information" rather "errors".

Let me go further in that the information theories are considering optimal compressibility of data used to transmit information. Interpolation, which is done in an algorithmic fashion is essentially "decompression" and not considered to add new information (or only adds a very small amount of information, that to specify the algorithm). In no case are the interpolated data points themselves considered new information. ***

*** at least in no theory that I am aware of, if thou doth protest, feel free to point me to a new theory:) Definitions are in fact tied to specific theories (their context) and each theory may define terminology in a differing fashion.

jabbr, thanks. I've got some further reading to do to try to understand this better, and probably won't have time today. So I'll see if I can get back to the discussion early in the week.

(Moved from the other thread:)

 

 

 

I disagree. The information between the original samples is already there, contained within the samples. Upsampling adds more samples, but not more information. It especially does not replace information that was lost during the sampling operations. That was lost in the anti-aliasing filter and can never be retrieved. The best you can do by upsampling is generate samples that match what they would have been if you'd sampled at the higher rate but used the lower rate filter. Sampling at the "1x" rate already captures all the information the filter passed. Sampling at "2x" is 50% redundant. You can discard every second sample and still have all the information that the filter passed. Ditto for upsampling after the sampling.

 

It doesn't amaze me that people get confused by this. It's non-intuitive. If it was easy, we wouldn't need basic fact education such as Monty's "Digital media primer" and "Digital show and tell" videos.

 

I do agree that in the real world upsampling has its uses, especially for reconstruction filtering at the DAC stage.

Perhaps this semantics, I am not sure.

Certainly, the potential to recreate the missing information is there in the 16/44.1 file, but in physical (and practical) terms, the information there is knly the static information that resides in the file.

Not until the file has been processed and recreated does that information actually exist. The recreation is never perfect due to practical, not mathematical considerations.

The upsampled file has more static information in it, in practical terms. (i.e. - the file is much larger, takes up more space, and without any processing, if plotted, would have far more data points / samples and produce a smoother plot.) Regardless of how I look at it, the upsampled file has more information. How useful that information may be is questionable.

Other than that minor point, which lay well be semantic, I agree with what you are saying.

Understand, that larger file to me represents more storage space, backup space, archival space, tranmission time, and ultimately, if a few bits go missing from it, the integrity of the music information it carries has actually been compromised. In fact, it is likely the file will not play.

The potential information that can be extracted from the information in the file is to me, only potential at that point.

Perhaps this semantics, I am not sure.

 

Certainly, the potential to recreate the missing information is there in the 16/44.1 file, but in physical (and practical) terms, the information there is knly the static information that resides in the file.

You are mistaking data for information.

Perhaps this semantics, I am not sure.

...

Not until the file has been processed and recreated does that information actually exist. The recreation is never perfect due to practical, not mathematical considerations.

 

The upsampled file has more static information in it, in practical terms. (i.e. - the file is much larger, takes up more space, and without any processing, if plotted, would have far more data points / samples and produce a smoother plot.) Regardless of how I look at it, the upsampled file has more information. How useful that information may be is questionable.

 

Other than that minor point, which lay well be semantic, I agree with what you are saying.

 

Understand, that larger file to me represents more storage space, backup space, archival space, tranmission time, and ultimately, if a few bits go missing from it, the integrity of the music information it carries has actually been compromised. In fact, it is likely the file will not play.

 

The potential information that can be extracted from the information in the file is to me, only potential at that point.

Paul, it is semantics, but to be clear, this is the definition of "information" according to information theory.

Consider this:

You have a 1mb FLAC stored on a 1Tb disc. You wish to backup the disc, let's say to a zip archive. How much "information" is in the ZIP archive (let's assume its about 1mb)

a) ?1 MB

b) ?1 TB

most people would say: a) 1 MB

Now you expand the zip to a brand new shiny 1 TB drive, how much information is on it:

a) 1 MB

b) 1 TB

Most people would say a) 1 MB.

Just the same, until the ZIP archive has been decompressed (and the new disc padded with lots and lots of 0s) it cannot be used.

So in information theory, the minimum compressed amount of data (+/- algorithm) required to recreate the "signal" is termed "information", regardless of the subsequently expanded, interpolated, padded, repeated etc etc etc size.

You need to reset your definition of "information" from "plain ol' bits" to "maximally and optimally compressed bits"

Sounds really good...

You are mistaking data for information.

Perhaps, but I do not think so.

A. Data only becomes information when it has meaning.

B. When a sample is created by interpolation between two data points, it becomes information about the signal.

C. The interpolated samples have meaning.

This of course is only true when the sample is actually interpolated, not when sample and hold is the order of the day. Also, it does not mean that the vast majority of the created smples will actually be used, as you have pointed out.

Yes, I can see it as a semantic issue too.

If you think of the information as being the original analogue signal to be digitised and later recovered, then upsampling adds no new information. The same signal can be recovered whether you upsample or not. (Assuming perfection, of course.)

If you think of information as being the digital samples, then upsampling adds information.