Secure ripping

[sbooth]

After some thought I arrived at the same conclusion as ye- but he beat me to it. With a damaged disc, if the drive is reporting the same data every time there is no way for the comparison ripper to tell the difference between good and bad data. The premise the ripper is working off of is that consistent data is correct data, where correct is defined as the data contained on the disc.

I think for a disc like the one you mentioned, one needs to examine the C2 (or some other parity bits) to determine that it is in fact damaged. Since the comparison ripper does not look at the subcode, it won't be able to tell. I am working on adding error-checking, but haven't gotten too far yet.

[ye]

Well, to be sure to have read the (raw) data as accurately as possible from the media is the first and essential step for further processing. So comparison ripping is a good thing to have!

As a next step an sophisticated error detection and correction would be nice. It should be possible to do this far better than a consumer CD player, as we have more computation power and do not have to do it real time.

But one thing I am not sure about: do we really have access to the "raw" data on the CD or has the CD drive some build in error correction logic that we can not circumvent? So do we have access to the unprocessed CD data?

[djb]

I read something about this somewhere.. isn't it that the C2 error correction is read by default and you need to turn it off in the software to get unprocessed CD data?

Max must be reading the C2 error correction from the drive when you do a comparison rip then i guess

By the way it is possible to rip on my iMac without C2 - running EAC in a virtual window works just fine. It's really slow so I don't use it often but it does work.

[sbooth]

I read something about this somewhere.. isn't it that the C2 error correction is read by default and you need to turn it off in the software to get unprocessed CD data?

Max must be reading the C2 error correction from the drive when you do a comparison rip then i guess

Maybe I was a bit unclear on what I meant- whether or not C2 is enabled by default, I'm not sure (I believe it is, though, but I haven't seen an OS X API for turning it off). I was referring to actually examining if C2 errors were detected, which currently isn't done. For a given CD sector I have access to:

• 2352 bytes of audio data
• 294 bytes of error flags (an array of 2352 bits, with ones where C2 errors were detected)
• 96 bytes of subchannel data
• 16 bytes of Q subchannel data

I am only slowly coming up to speed on CD error correction, but as best I can tell it is a bit more of an art than a science. While the sectors have a CIRC (and more) for error checking, the decoding/processing and actual error correction is drive-dependent.

I believe that examining the error flags and determining if any C2 errors were detected could give a better idea of the state of the CD.

[ye]

Hmm,... so this gives us only the result of some black box error correction plus an indication which bytes possibly weren't corrected accurately. No chance to do a error correction of our own with this data.

But how about this feature of cdparanoia that they claim to be an error correction. Is there any documentation what is done there, I have not found anything up to now?

[sbooth]

Hmm,... so this gives us only the result of some black box error correction plus an indication which bytes possibly weren't corrected accurately. No chance to do a error correction of our own with this data.

But how about this feature of cdparanoia that they claim to be an error correction. Is there any documentation what is done there, I have not found anything up to now?

The only part of cdparanoia I really understand is the overlap (jitter) correction. As far as the rest of its magic goes, I haven't found any solid documentation either.

I am going to experiment with the error flags and see if I can glean anything from them in my testing.

[ye]

I am going to experiment with the error flags and see if I can glean anything from them in my testing.

One nice thing, that should not be too much work, is to report how many (percentage) there are (per sector per track, per rip,... perhaps also graphically?). This would report the percentage of defective bytes, or more correctly, the percentage of bytes the error correcting black-box algorithm can not "guarantee" to be correct.

As far as I see, you can not use the data you mentioned for real error correction (in the sense of error correcting codes). But one thing that could be done, as an option, could be to replace the bytes flagged as faulty by a value interpolated from the correct neighboring bytes.
But I thing this should be done under user control or a t least some intelligent replacement, e.g. only if the faulty value differs much from the interpolated.

This would be no error correction, but perhaps could help to get a better audio quality of damaged disks (avoid clicks etc).

[Hermie]

The ClickRepair guy probably knows a lot about this. His app does miracles on my vinyl recordings.

It's here:

http://wwwmaths.anu.edu.au/~briand/sound/

[macfeller]

I'm sure it's not possible or you would have done it, but I'd appreciate knowing why the whole "filling the cache" thing can't somehow be combined with cdparanoia: fill cache then call cdparanoia, take the part in cdparanoia where the cache is filled and "change" that setting to something tiny, etc.

I apologize if this is discussed elsewhere. I am too clueless to truly know what I read on such discussions (e.g. hydrogenaudio).

[sbooth]

I haven't tried the same disc using the paranoia settings so I don't know about that yet.

Out of curiousity how did paranoia fare with the damaged disc?

[sbooth]

I'm sure it's not possible or you would have done it, but I'd appreciate knowing why the whole "filling the cache" thing can't somehow be combined with cdparanoia: fill cache then call cdparanoia, take the part in cdparanoia where the cache is filled and "change" that setting to something tiny, etc.

I think it's possible in general (it's been done in linux), but it isn't very doable on Mac OS X. Paranoia works (as best I understand it) by reading a sector and its neighbors, then checking for overlap in the sectors, aligning them as needed, then magically correcting errors it finds in the data using parity check bits. If you have a drive that caches CD audio, paranoia will not be as effective because when it detects an error and re-reads a sector, that sector will come from the drive's cache. Obviously if you suspect errors, you want to hit the physical media and not the cache of the last read.

SBC-3 specifies ways to force reads off the physical media (FUA bit for READ 10 or READ 12), but currently on OS X there isn't a way to use these without writing a device driver from scratch (as far as I can tell).

So, the problem is, ripping would be unbearably slow if in between each sector read (a sector is only 2352 bytes of audio) you had to read 2MB+ of data you don't care about just to fill the cache with "junk", before re-reading the single sector. If each sector was retried 20 times, that would be 40 MB+ of extraneous reads just for cache clearing.

This is why I wrote the comparison ripper- it rips large chunks at a time, clears the cache, and re-reads the same chunk again. The assumption was that damaged disc areas would generate inconsistent data. This is turning out to be an incorrect assumption, but the comparison ripper is a great start for adding C2 checking and parity checking as well (EDC).

[sbooth]

One nice thing, that should not be too much work, is to report how many (percentage) there are (per sector per track, per rip,... perhaps also graphically?). This would report the percentage of defective bytes, or more correctly, the percentage of bytes the error correcting black-box algorithm can not "guarantee" to be correct.

I've implemented this as a test using the basic ripper. I will run trials on damaged discs and see how accurate the C2 reporting is.

As far as I see, you can not use the data you mentioned for real error correction (in the sense of error correcting codes). But one thing that could be done, as an option, could be to replace the bytes flagged as faulty by a value interpolated from the correct neighboring bytes.
But I thing this should be done under user control or a t least some intelligent replacement, e.g. only if the faulty value differs much from the interpolated.

I think this is already the way C2 works on drives! You've hit the nail on the head, although from my reading it seems the way this is actually accomplished is drive-dependent. I think the variance arises from the way certain higher order polynomials are approximated, but I am still learning in this area.

This would be no error correction, but perhaps could help to get a better audio quality of damaged disks (avoid clicks etc).

I can still access the subchannel data, which contains some data which can be used for error checking and correcting. More on this to come.

[ye]

I can still access the subchannel data, which contains some data which can be used for error checking and correcting. More on this to come.

I looked around on many sites to get infos about the data structure on the Audio-CD. The info is often contradicting I fear I have to have a look in the Red Book if I have some time and get access to it

But up to now I am quite sure that the parity data is NOT provided in the subchanels. The most convincing argument is the lack of amount of information:

To 24 data bytes there are added 4 bytes of parity information in the first encoding step (in most places called C1 encoding), and 4 bytes of parity information in the second encoding step (in most places called C2 encoding).

So for the 2352 bytes of audio data per sector you get additional 784 bytes of parity data, you must have access to, to do your own error correction. This amount of additional bytes does not appear in your list of available data.

294 bytes of error flags (an array of 2352 bits, with ones where C2 errors were detected)
96 bytes of subchannel data (*I assume this is a Typo and is 98*)

This is just half of this data. So if we are lucky we can do one layer of decoding (of two encoding steps) ourself. But more likely , taking in account the naming of these data, it contains other information.

[macfeller]

Thanks for the explanation re: difficulty with "just fill the cache with junk" strategies.

[sbooth]

I looked around on many sites to get infos about the data structure on the Audio-CD. The info is often contradicting I fear I have to have a look in the Red Book if I have some time and get access to it

If you don't want to spend $$ on the Red Book http://www.ecma-international.org/publi ... ma-130.htm is almost the same.