-------------------- AmigaOS 3.1.4.(1) project ----------------------- DiskDoctor 2.52 (20.1.2018) - First beta test release to escape from the labs. DiskDoctor 2.53 (20.1.2018) - Progress display for complete block scan erased everything it printed right after having printing it. Ouch. It should have only erased the remainder of the line following what was printed. - Block damage simulation for the root directory only overwrote a single byte, but it should flip a couple of bits, to make sure that the contents are different from what there was before. - Decoding the BCPL-ish "*x" escape code now checks how many characters (digits or the letters a through f) it managed to convert. If it failed to convert even one, it now prints the "x" or "X" following the "*". DiskDoctor 2.54 (21.1.2018) - Added more specific error codes and messages which replace ERROR_OBJECT_WRONG_TYPE and ERROR_OBJECT_NOT_FOUND within the directory tree structure management functions. - The number of blocks per volume was miscalculated, because it did not take the number of sectors per block into account. This had repercussions on the proper detection of the root directory, which would render all the files of a volume orphans when this was not really the case. It also would result in too much memory to be allocated for keeping track of the blocks found. Finally, all the tests which checked if a block number was within valid bounds would produce unsafe results. DiskDoctor 2.55 (21.1.2018) - When recording the names and comment texts of directory entries we now make sure that this information is not obviously invalid. If there are supposed to be more characters stored than the original data structure could contain, then the respective text is truncated to the maximum length permitted. Each invalid character found in the name is replaced with a '#'. Empty names are replaced with a prefix identifying the type of directory entry (e.g. "file", "directory", etc.) followed by the number of the respective directory entry header block. DiskDoctor 2.56 (25.1.2018) - Now properly files orphaned directory entries in the root directory, if their respective parent directories are not valid. DiskDoctor 2.57 (26.1.2018) - If the COPY mode is not in effect, using the TO option will show an error message. The point is that you should be aware of the consequences of using the COPY mode (e.g. disk space requirements, etc.) and not have it triggered by accident merely by using the TO option. - If the VERBOSE switch is in effect, the copy operation will now print more information about the restrictions placed upon it, e.g. what may be copied, and if the PRETEND switch will make it skip creating files and directories. - All sector and block number variables are now unsigned 32 bit integers. - A stale pointer is now cleared immediately after closing the disk device driver, before reopening it. Reopening might fail, and that stale pointer could have been trouble. DiskDoctor 2.58 (28.1.2018) - Added more runtime debugging functionality, which includes diagnostic messages and assertions. These are baked into the command right now and will send output to the serial line via kprintf(). Careful if you are using the built-in serial line for something important. I recommend you capture any debug output with the Sashimi command. - BCPL string copy operations now observe the target buffer size limit, not unlike strlcpy(). - Added stack size checks in all recursive functions. - Added tests to determine if an unvisited directory entry without a matching parent directory is really an orphaned entry. If it's not, then there's something seriously wrong. - Extended the same paranoid parent directory checking to the function which looks at all unvisited directory entries. DiskDoctor 2.59 (30.1.2018) - Improved handling of corrupt/invalid long name comment strings. - Soft link targets may contain path separator characters, which requires a different validation than what works for file and volume names. - Length limit violations for directory entry and volume names now lead to more specific information being printed as to what is wrong (empty, too long). - If the block contents scanner finds directory entries which seem to have an empty name on a long name volume, it now reports them as likely being "short name" directory entries. - The recursive directory tree scanner needed some 848 bytes of stack space per directory level, which means that it exhausted the safe stack space after six levels of recursion. New code now dynamically allocates the working memory needed for the scanner, and as a last resort also watches how much safe stack space is still available. If there are fewer than 4000 bytes of safe stack space available, the scanning process is aborted with a matching error message. DiskDoctor 2.60 (30.1.2018) - Directory validation could get stuck in an infinite loop. DiskDoctor 2.61 (30.1.2018) - Queued disk read requests are no longer aborted when Ctrl+C is used during the block scanning stage. They are allowed to finish, which might take a few seconds. I suspect that not all device drivers permit read commands in progress to be cancelled (trackdisk.device, scsi.device, etc. certainly have no such problem). - Memory accounting could read size information from data structures after they had been freed. Ouch :-( - Added 32 bit multiplication overflow checking to the various volume parameter juggling operations. This is "just" a precaution, as most of these volume parameters were never even close to causing overflows in the 1990'ies. DiskDoctor 2.62 (1.2.2018) - FFS data blocks are now counted while their associated file header and extension blocks are being scanned. Previously, the number of file data blocks used would remain zero. - Now correctly processes the hash tables of the directories it encounters when copying the file system tree. The hash tables are now part of the state information which this recursive function uses, which saves the trouble of reallocating the hash table copy for each directory. - Added the WHY switch to the command line template. Whenever a copy operation is skipped, the WHY switch will make Disk Doctor print a short explanation. - Updated the documentation. DiskDoctor 2.63 (2.2.2018) - Added more sanity checks with regard to the volume parameters. For example, the number of reserved blocks must be smaller than the total number of blocks per volume. Additional arithmetic overflow tests have been added as well. - The first sector of a volume is now read only if there is at least one reserved block at the beginning of the partition, which ought to contain the file system signature. Whatever is found is now taken as is, rather than ignoring unsupported file system formats. - If the file system format is unsupported, Disk Doctor now states this in a warning message and also indicates that it will fall back to DOS\0 (OFS) format, or DOS\1 (FFS) format if the volume is too large for OFS format. - Added a more informative volume size test which checks if the volume is larger than the maximum number of blocks available for an OFS format volume. This is not necessarily a fatal error, which is why the corresponding error message in the root directory check has been removed. - Updated the code documentation and the command documentation itself, which now includes details on the environment values which Disk Doctor looks at. These environment variables are useful for testing. - Replaced the asynchronous read request shutdown procedure again. Let's see if it works this time. DiskDoctor 2.64 (2.2.2018) - The file system signature information is no longer printed just in hexadecimal form, but also in (mostly) human readable form. For example, 0x444F5300L is now also shown as 'DOS\0'. DiskDoctor 2.65 (3.2.2018) - Fixed the octal number digit generation. - Added more buffer bounds checking code. - Correctly closing the DABI file was recorded as a failure. Fixed. - If you abort the copying operation, the code which fixes up the directory metadata can now be skipped. Hence, the whole process should conclude much faster. - Added stack size checks prior to all sorting operations. - When preparing the "inventory lists" for the DCFS directory cache lists and their respective associated directories, the metadata now covers everything that goes into the directory cache lists, and not just the block number, directory entry type, size and name. This both allows for a more accurate comparison, and for reconstructing the cache lists, should the need arise in the future. - Errors reported by zlib are now displayed in context, if possible. For example, failing to wind up the last compression operations when a DABI file is closed is now considered an error and the incomplete DABI file will be deleted. DiskDoctor 2.66 (4.2.2018) - Added code for detecting overflowing addition of unsigned 32 bit integers. Actually, the function calculates the carry of the sum. - With the PRETEND switch in effect, Disk Doctor no longer insists that the COPY mode has to have a valid TO parameter, which would have been ignored anyway. DiskDoctor 2.67 (7.2.2018) - Preparations begin for allowing the copy operation to reconstruct hard links. This includes finding the hard link targets during the examine stage. DiskDoctor 2.68 (10.2.2018) - Renamed the "WHY" switch to "EXPLAIN". - The "VERBOSE" switch always had the effect of also enabling progress reports, as if the "PROGRESS" switch had been used. However, there was very little to be learned by using the "PROGRESS" switch only. This has been changed, i.e. you will now see general progress reports on what is going on while the examine/copy operations are being performed. - Hard link reconstruction, as part of the copying operation, is almost ready for testing. DiskDoctor 2.69 (10.2.2018) - Hard link reconstruction, as part of the copying operation, now actually works. The reconstruction is now broken up into two passes. The first pass reconstructs the file system directory structure, ignoring hard links altogether. When this pass is successful, the second pass will repeat the same operations, but this time only pay attention to the hard links. This approach should make sure that the directory entries which the hard links are attached to already exist. - The DABI file now stores information about the directory entry blocks referenced by hard links. - The soft link target path names were never properly recorded in the directory tree data structures. Fixed. This means that soft link reconstruction now works for the very first time. DiskDoctor 2.70 (11.2.2018) - When reading from DABI files, the block numbers are now checked for validity. This includes testing for possible numeric overflows, too. - Disk Doctor now requires volume block sizes to be a multiple of four. The required minimum block size is now 512 bytes, regardless of the sector size. - Updated consistency checking for DCFS directory cache block contents, verifying that the lengths of names (can be only up to 30 characters in size) and the contents of comment text conform to the size of the block buffer. - Fixed the link cycle detection code. It was broken because of a missing "~" character in the code which deals with removing block attributes from the in-memory database. - Made sure that Disk Doctor still builds correctly if the DEBUG option is disabled, and that no debug code remains if that option is in fact disabled. DiskDoctor 2.71 (11.2.2018) - Prior to copying, the destination is now examined and error or warning messages are printed if it appears to be a bad fit right now. If the destination is not a directory or volume, or if the destination is not writable, or any other kind of file system error occurs when accessing it, Disk Doctor will print an error message and exit. If everything else checks out, but there is very little space left on the destination, a warning message will be printed. DiskDoctor 2.72 (17.2.2018) - Cleanup in preparation for the "repair" operation. DiskDoctor 2.73 (23.2.2018) - Preparations for checking the validity of the bitmap contents. - The bitmap block check did not stop after having looked at exactly as many blocks as the bitmap should have consisted of. It just kept on going, looking at every bitmap block in the tables. DiskDoctor 2.74 (24.2.2018) - Added reconstruction of the volume bitmap. The reconstruction is eventually compared against the contents of the bitmap, as read from the volume. Any differences found suggest that the bitmap may have to be rebuilt as part of the repair and reconstruction operation. DiskDoctor 2.75 (24.2.2018) - If the volume is in need of validation, or other kinds of errors are detected, the examine operation will end with Disk Doctor exiting with the return code set to 5 (warning). This only goes for the examine mode, i.e. the copy mode or the still to be implemented repair mode will continue. The idea is that you could use "diskdoctor examine hd0:" followed by "if warn" in a script file, just to check if a volume is probably sound. DiskDoctor 2.76 (25.2.2018) - If a block is recorded in the bitmap as "used", Disk Doctor now also makes sure that the block is marked "reachable", so that after loading a DABI file the bitmap may be reconstructed as part of the repair process. DiskDoctor 2.77 (25.2.2018) - The need to repair the contents of a directory hash table is no longer tracked as an attribute of the directory, but as a defect. Directory entries which were added to the root directory are now identified as such, so as to make a repair operation easier later. - How many bytes are stored in the data blocks associated with a file are now added up. If that number is greater than or equal to 2 Gigabytes, the file will be marked as defective. This type of defect is different from a file header merely reporting that a file size is than or equal to 2 Gigabytes. It catches the hopefully very rare case of files larger than or equal to 4 Gigabytes. This still needs more work, as the code in examine_all_files.c/examine_file() does not yet check the edge cases of files being larger than 2 Gigabytes. - Reworked the defect reporting for links, which now properly discriminates between the link target being unsound and the type of directory linked to being the wrong type of entry. DiskDoctor 2.78 (3.3.2018) - The copy operation now only uses a single counter to determine how many more bytes it has to copy. That counter also takes into account if blocks have to be skipped. Their respective byte sizes are subtracted from the counter. This is the simplest possible "backstop" for the FFS managing to store too much data in a file and dealing with the fallout. The FFS may show undefined behaviour when writing more than 4 Gigabytes of data to a file. - When examining a file for defects a single byte counter is now used, too, to determine if the file is shorter than it was supposed to be. Again, this is a backstop for the FFS storing more than 4 Gigabytes of data in a file and dealing with the fallout. - Dropped the "directory entry was added to the root directory" flag, because a repair operation will have to deal with these directory entries differently. - Defects found during the examination are now themselves examined more closely regarding their weight and impact. Defects are considered relevant for reporting only if they concern those parts of the file system which are reachable through the root directory. If they are reachable, a defect is considered serious if it impacts the integrity or the reliability of the volume, such as an invalid name, directory cache blocks whose contents differ from their respective associated directories, corrupt file data blocks, etc. DiskDoctor 2.79 (3.3.2018) - Printing verbose information about the bitmap contents could lead to trouble with volumes without a proper root directory or bitmap structure. Fixed. - When examining the contents of the root directory, verbose mode now shows a bit more detailed information about it, which includes the volume name, the disk initialization date/time, the last root directory change date/time and the last volume change date/time. DiskDoctor 2.80 (3.3.2018) - Removed references to the recovery/salvage operations. What remains is the currently unimplemented repair operation, which is being prepared. DiskDoctor 2.81 (3.3.2018) - Added more arithmetic overflow tests, coupled with fall-backs for performing the troublesome operations using 64 bit integer math. DiskDoctor 2.82 (7.3.2018) - Replaced the duplicate detection method used by the directory contents scanner. We now use a balanced binary search tree rather than building a list of names, sorting it and comparing neighboring entries. The search tree solves this with more effort, but it can be reused by the repair operation when picking new names for the duplicates. This is where it will become really useful. DiskDoctor 2.83 (20.3.2018) - An invalid file extension block could cause the copying operation to loop, without allowing the user to cancel it. Fixed. - Replaced the splay tree which was used solely for keeping track of hard link target block numbers with the red-black tree which is used for the name list, too. The red-black tree likely scales better than the splay tree, but I was unsure if it would be a good idea to bring out the big guns so early. DiskDoctor 2.84 (29.3.2018) - The root directory's "last volume change" field is now only examined if the file system is in FFS format, as this field does not exist for OFS format. - The DABI file now contains information about the volume name and the volume initialization date and time. This information will be useful for the repair functionality. The "repaired" volume will be modified to feature a initialization time which will differ from the original by one tick in the future. Volume name and initialization time together form what should be a unique identifier for the volume. DiskDoctor 2.85 (3.4.2018) - Added support for the ACTION_GET_DISK_FFSM/ACTION_FREE_DISK_FSSM packets. This makes finding both the file system responsible for the volume and obtaining the disk parameter much easier, if the file system in question supports these packets. - Preparations begin for implementing the volume repair operation. This includes checking the root directory, to see if it can be repaired (this Disk Doctor will not raise Lazarus). - In verbose mode the examine operation how reports how many unreadable blocks were found and shows a percentage figure (how much of the volume is unreadable). DiskDoctor 2.86 (17.6.2018) - Added more debug output for the Disk Doctor startup, before anything interesting could happen ;-) - Changed the tests for NSCMD_TD_READ64/TD_READ64 compatibility. The error code returned has to be 0, and the number of bytes read has to match the sector size. DiskDoctor 2.87 (18.7.2018) - Added the complete SCSI direct I/O functionality implemented for use with the updated FFS and CrossDOSFileSystem, courtesy of Thomas Richter. Thank you very much! - Disk Doctor now defaults to use the TD64 commands when trying to access data which resides beyond the 4 Gigabytes of the medium. A fall-back to the equivalent NSD is present. - The flags which control whether disk access should use direct SCSI I/O, or refrain from using the NSD trackdisk.device commands are now respected. The same goes for the related "superfloppy" mode which forces the device's current sector size to be retrieved and used in place of the default value set for it. DiskDoctor 2.88 (21.7.2018) - Added command line override parameters for the disk geometry check, the maxtransfer value, the buffer memory type and the disk access mode. Updated the documentation accordingly. - If the disk geometry is checked, the parameters (sector size, number of cylinders, number of surfaces, number of sectors per track) are now updated accordingly at the earliest possible stage for the disk access functions to validate and eventually use them. DiskDoctor 2.89 (22.7.2018) - The FFS uses 25 bitmap blocks (as stored in the root directory), with the 26th block being the bitmap extension, whereas the original Amiga ROM file system (OFS) would use all 26 block entries. This affects auto-detection of the file system flavour which for some storage devices, owing to their size, cannot be OFS. - Moved the translation of the SCSI status into a trackdisk.device error, as well as the translation of the scsi_Actual field into io_Actual into the finish_disk_io_request() function, which removes duplicate code. - The cylinders/heads/sectors per track information retrieved by TD_GETGEOMETRY or its SCSI counterpart is now used by Disk Doctor, instead of relying on the respective values which were previously obtained from the device's DOS environment vector. - When reading the DABI file the volume initialization date is now processed correctly. It used to parse the wrong text and should have only looked for the number of days, minutes and seconds. The error reported for the DABI file was misleading. - Added more sector size sanity checks which test for an overflow conditition and if the sector size is a power of two. - The debug output can now be sent to STDOUT/STDERR or appended to a given file. It is no longer necessary to use a program such as "Sashimi" to capture it. Where the debug output ends up is controlled by the "diskdoctor/debug-file" environment variable, e.g. "set diskdoctor/debug-file stdout" for standard error output. See the DiskDoctor documentation for more information. - If the mass storage device driver does not support SCSI commands, Disk Doctor now aborts the examination at the earliest possible stage. DiskDoctor 2.90 (27.7.2018) - Disk sector sizes must now be a power of two. While in theory sector sizes could be a number as long as it can be divided by 512 in practice the requirements are even stronger. - Unless otherwise specified, the size of the buffer used for reading and writing is now large enough for an entire track to be read. How large a track is will be figured out dynamically as needed. - The number of sectors per track was reported as the number of blocks per track, which is not correct. The DABI format now reflects this. - The DABI file no longer requires the numbers of the lowest cylinder, the highest cylinder and the number of the first sector to be present. This data would nail down the position of the partition, while only the size has to be constant. DiskDoctor 2.91 (28.7.2018) - The default size of the buffer used for reading the disk during the examine stage now always allows for complete blocks to be read. Previously, there could be trouble if the number of sectors per track were not a multiple of the number of sectors per block. - The DABI file now includes the number of cylinders, which must match the same number of cylinders for the volume which it is used with. Instead of fixing the position of the partition by recording and then verifying the highest and lowest cylinder numbers, respectively, the size parameters are being fixed. - Added more debug output to the DABI file processing code. DiskDoctor 2.92 (28.7.2018) - Debug output now includes the partition parameters in readable form, rather than as compounded quantities. - Error output now really goes to stderr if pr_CES == NULL, rather than to Output(). - When scanning all the blocks of a partition the "position" information now shows the block number ("key") rather than the sector number. This may be a little more useful than it was before. - If the destination of a copy operation does not yet exist (e.g. in "diskdoctor copy df0: to ram:destination verbose"), Disk Doctor will create a directory (e.g. "ram:destination") before it begins the copy operation. DiskDoctor 2.93 (28.7.2018) - When comparing block types the DABI processing code did not always mask out the attribute information stored along with the type, and so did other parts of Disk Doctor. Ouch :-( This led to the creation of unusable DABI files, and worse, rendered deleted files wholly unrecoverable. Thanks go to Thomas Klein for producing the test cases which made it possible to find the problem. DiskDoctor 2.94 (28.7.2018) - Fixed a bug in the assert.c code which had the effect of writing the file and line number information to serial output when it should have gone to a file instead. - Moved the detection of the file system signature, so that it occurs at the earliest possible time. This change also allows you to override the file system type through the command line options, even if the file system information came from a DABI file. Previously, you could override the signature only directly after the first block had been inspected which never happened if a DABI file was read. DiskDoctor 2.95 (29.7.2018) - Replaced all the inline bit mask and bit test operations with macros whose names speak of the purpose of the respective operation performed. It was getting really hard to check just by looking at the code what was going on, and to which end. DiskDoctor 2.96 (30.7.2018) - If possible, error messages printed now also show the explanatory text for the numeric error code which prompted the error message to be shown. Previously, you might have been forced to use the "Why" shell command to show the error message. - Error messages produced by the copy operation now go to the standard error output. DiskDoctor 2.97 (30.7.2018) - Error messages are now printed during the examination process only if progress reports were requested. DiskDoctor 2.98 (30.7.2018) - If a file, directory or file extension block does not report its own block number as the one which it was retrieved from, this is no longer reported as and treated as an error. A warning is printed instead. - If a file appears to contain more data than the data blocks associated with appear to provide this no longer counts as a defect. DiskDoctor 2.99 (31.7.2018) - Replaced the PROGRESS switch with QUIET, which has the exact opposite effect. Disk Doctor now defaults to printing progress messages, and if you do not want to see them, you would need to use the QUIET switch (and refrain from using the VERBOSE switch at the same time). DiskDoctor 2.100 (4.8.2018) - Changed "ENVF_NONSD" to "ENVF_DISABLENSD". - Replaced the PutMsg() in the asyncio code with SendPkt(), like it's supposed to be. - Made the use of HD_SCSICMD a little safer by setting io_Actual to 0 for all cases before the command is sent to the driver. - Added more sanity checking to the TD_GETGEOMETRY results. All the members of the C/H/S tuple now need to be > 0 for any individual member to even checked. - Direct SCSI access for read/write operations is now only used if needed. However, the TD_GETGEOMETRY SCSI-only replacement is still used if SCSI access is enabled, rather than using the TD_GETGEOMETRY command. - The progress messages now show how many steps are involved in the entire process e.g. "Examining files for defects (step 4 of 11).". Yes, these go to eleven. - Individual progress reports now show, if possible, how many more entries need to be processed before the current step will be completed. Note that this does not work with the directory tree search step since the very purpose of this task is to figure out how many entries there are. But this step produces so much output anyway, it's hard to overlook that something is happening. DiskDoctor 2.101 (5.8.2018) - Added the LOGFILE/K argument and the code to support it. - The use of Printf() and FPrintf() has been modified to make it easier to tell the difference between output which should go into the logfile and which should be omitted. DiskDoctor 2.102 (6.8.2018) - Simplified the progress and verbose output operations further so that their respective output always goes into the log file, if enabled, but will be not be printed if the QUIET option is used or the VERBOSE option is omitted. - Added a bit of extra paranoid sanity checking to the io_Actual value returned by read/write operations. In the odd and unexpected case of the device driver reporting successful execution and that more data was processed than called for, the IORequest error code is changed to IOERR_BADLENGTH. The hope is that the device driver didn't actually read/write more data than what was called for... - Updated the documentation, specifically the section which details the use of the LOGFILE option. DiskDoctor 2.103 (6.8.2018) - A duplicate integer parameter passed as the format string to VerbosePrintf() broke the log file output. Fixed. - The log file is now created as MODE_READWRITE since MODE_NEWFILE cannot be switched to MODE_OLDFILE later due to a RAM-Handler bug. - More robust percentage figure calculations for the number of blocks used and the number of unreadable blocks. - The log file now contains information on the total running time. DiskDoctor 2.104 (6.8.2018) - Replaced duplicate printing of warning/error messages (once for the console, and then again for the log file) with a more robust solution. - 64 bit integer fallback code for percentage, etc. calculations promoted to the default approach. - The block scanning stage progress information is now dynamically adapted to show only the respective number of days/hours/minutes/seconds remaining. - The log file now also includes the command's exit code (success, warning, error, failure). DiskDoctor 2.105 (7.8.2018) - Replaced the 64 bit integer multiplication algorithm with something much simpler. The 64 bit integer division algorithm now returns the remainder, which allows the function which converts the number into text to be simplified. - Updated the documentation for the LOGFILE option. - The signal_exception.o file is required by the repair_volume.o file and need not be built yet. - The DELETED and ORPHANED filters can be used simultaneously with the COPY operation now. - Added more specific explanations as to why a directory entry will not be copied. DiskDoctor 2.106 (7.8.2018) - When performing a copy the modification time of the original file is now always preserved unless the copy had to be deleted. Zero length files, or files which cannot be copied completely (the Close() function returned an error) will now be taken care of correctly. - The copy operation now will just keep going even if minor problems crop up. Only major issues, such as running out of memory, will make it stop. - The copy operation no longer prints warning messages for minor problems. If any messages need printing, these will be error messages which led to the copy operation to be aborted, or warnings about truncated or corrupted file copies. DiskDoctor 2.107 (7.8.2018) - The copy operation could fail to unlock directories/files after testing if the respective directory entry was already present. Fixed. - The copy operation no longer warns about file copies being truncated or corrupted if the user specifically asked for only the damaged files to be copied. - Added detailed error messages to the copy operation where these specific errors were previously not reported in context (what went wrong, which directory entry/path was involved, etc.). - Corrected a ChangeMode() invocation which could have led to a crash... DiskDoctor 2.108 (10.8.2018) - When writing to the debug file, the output is now line-buffered. If the debug file already exists a line feed will be added before any further output is written to it, just so you can see where the next "session" begins. - When reading from FFS data blocks the test to check if the block had already been visited could trigger an error condition precisely if it had not been visited yet under one of those very sad edge conditions which seem to crop up at the most unexpected time. Fixed. This should solve the mystery of files getting created by the copy command which remain empty in spite of the fact that that the respective original does contain recoverable data. - Reviewed the code which works very similar to the odd FFS data block slip-up and made some changes for better readability, and if possible, better error reporting. DiskDoctor 2.109 (13.8.2018) - The debug file will not be created unless the debug mode is enabled. - Fixed an undetected NULL pointer argument used by the copy process progress messages. - Removed a handful of unused variables. - The copy process now stops copying when the break signal is received right after the file copied has been closed. - If possible, the copy process returns the error code which caused the process to be aborted so that it may be displayed properly later. - Added slightly more consistent error reporting for hard link creation during the copy process. - Simplified the copy process progress information output to save space. Also, instead of just stating that a file would be copied, we now show how large the file is, which should be more useful. - When reading the DABI file orphaned directory entries are now always associated with the root directory, just to be sure. - The examine operation now makes sure that all orphaned directory entries will be associated with the root directory. This is necessary for the recovery operation to work. DiskDoctor 2.110 (14.8.2018) - Added progress information messages to the copy operation which state how much of a file has been copied so far, and for how much longer it will take for the file to be copied completely. Note that these messages will only appear it it takes more than 1-2 seconds for a file to be copied. - Problems which crop up during the file copy operation and which result in blocks getting skipped are now described in more detail if the VERBOSE switch is in effect. All of this information will be available in the log file, of course. DiskDoctor 2.111 (15.8.2018) - We now try to show exactly which AmigaDOS file system device is associated with the drive name the user picked. For example, "diskdoctor examine sys: verbose" may show the note that "Drive "sys:" is really "WB_2.x:". - The size of the file I/O buffer which is used when copying files and when reading/writing DABI files is now configurable. It used to be 8192 bytes (total). - Simplified the progress message output to save space. - If the log file already exists and is not empty, two line feeds will be written to it before the next log data is stored in it, so that you can tell the different sessions apart more easily. - When printing the root directory name we now try to make unprintable characters visible for display. If there are any unprintable or reserved characters (":" or "/") to be found in the root directory name a warning to this effect will be printed. - Individual error messages concerning the state of the bitmap are now only printed in verbose mode. There might be a lot of those errors and we don't want to flood the console with these messages unless it's necessary. - Changed the placeholder text for directory entries which lack a name. The text now reads as "file#123456", for example, whereas it used to read as "file-001E240" before. The change is necessary because the copy operation might just mistake the "-0" as a hint that a duplicate file exists because the OVERWRITE=keep option had been used and end up corrupting the name when making another copy. DiskDoctor 2.112 (16.8.2018) - The copy operation now prints a summary of the number of directory entries copied along with the number of bytes which were copied. If the PRETEND switch was used, these figures will be printed with a note that the values are only a simulation of the actual operation, which might end up copying fewer data than expected. - Cleaning up after the low level device access code is now more paranoid in avoiding to close the same device driver twice. - Added the SINCE and UPTO arguments which can be used to restrict the directory entries to be copied. The respective date and time of the last change will be compared, and if the entry is older than what the SINCE parameter says, or if it is newer than what the UPTO parameter says, then it will not be copied. The parameter to the SINCE and UPTO arguments can either be a date (e.g. "Yesterday", "16-Aug-18) or a time (e.g. 10:19:35"). If a time is given, then the date is assumed to be the current day. If a date is given, the corresponding time will be set to 00:00:00 of the respective day. You can use both SINCE and UPTO together, or you can use just one of the two. - Added the MINIMUMSIZE and MAXIMUMSIZE arguments which can be used to restrict the files to be copied. Prior to copying the respective file size is compared against the minimum/maximum size required. If the file is smaller than the minimum size, or if the file is larger than the maximum size, it will not be copied. Note that the decision is made based upon the size of the file known before the copy operation begins. For example, if the copy operation ends with fewer data copied than expected, Disk Doctor will still keep what it copied, even if the amount copied was smaller than what the MINIMUMSIZE argument required. The MINIMUMSIZE and MAXIMUMSIZE arguments only have meaning for files to be copied and do not apply to directories and links. You can use both MINIMUMSIZE and MAXIMUMSIZE together, or you can use just one of the two. - If the SINCE or UPTO options are used, the information printed by the copy operation for each directory entry now includes the creation/last change date and time. DiskDoctor 2.113 (17.8.2018) - The copy procedure now prints the block number ("key") of the respective source file, directory or link. This should make it easier to tell what was actually copied if, for example, deleted files or directories which share the same name with a "normal" file, are being copied. The names will be the identical but the keys will be different. - In verbose mode the copy procedure will now print a short summary of each individual file copy operation, noting how many bytes were actually copied (the file may have been truncated either during writing or reading). This information also goes to the log file. Again, the size information should make it easier to tell different files apart which share the same name. - Disk Doctor now calculates the SHA-1 checksum for each file being copied, with the checksum being calculated over what was actually stored in the output file. The checksum is displayed only if the VERBOSE switch is used, otherwise it only ends up in the log file. Again, the checksum should make it easier to tell different files apart which share the same name. Note that SHA-1 is used because it is easily the fastest algorithm for the job which should still work well enough to yield different checksums for files of identical size whose contents differ only by as little as a single bit. - When examining all directories, the block number ("key") of the respective directory was not necessarily displayed correctly. Fixed. DiskDoctor 2.114 (17.8.2018) - The log file now contains detailed information on which respective data and file extension blocks are being read during the copy operation. - Added the ALL switch which has the effect of copying everything found on the source volume, both the deleted, the orphaned and the neither orphaned nor deleted directory entries. This allows you to copy the known contents of the volume in one single pass. DiskDoctor 2.115 (17.8.2018) - When the copy operation is in progress, the type of object, the key (block number) and the modification time of the object will now always be printed. - When checking what the soft link target might reference, we now prevent the "please insert disk" requester from appearing, if it should show up. This is just to make sure that the show goes on if the soft link should reference a currently unavailable device or volume. - The SINCE and UPTO parameters can now contain complete date/time specifications if you put them into double quotes, e.g. ""Yesterday 13:32:08". - The format used for printing date and time information, as well as when processing the SINCE and UPTO parameters is now configurable, which should help writing portable scripts that utilize Disk Doctor without running afoul of locale-induced date and time format issues. DiskDoctor 2.116 (17.8.2018) - The copy operation now prints if the file being copied is orphaned/deleted or damaged. DiskDoctor 2.117 (19.8.2018) - Empty SINCE and UPTO parameters are now rejected as invalid. - The overwrite check could call Examine() with a ZERO lock. Fixed. - The in-memory directory entry database now sorts entries differently which share the same name. Files which are neither orphaned nor deleted are now placed closer to the beginning of the list than entries of the same name which are deleted or orphaned. The idea is to make it safer to use the "examine copy all overwrite=keep" procedure. With this combination of options, a "sound" file will be copied first, and all orphaned or deleted variants of it will follow next. - Directory entries which were marked as being unreachable (through the root directory) could be mistaken for being merely "deleted" if their respective parent directories were valid, even if these parent directories were orphaned directories. New code has been added which builds a comprehensive set of orphaned directories. This set not only marks a directory as being orphaned if its parent is invalid, but also if its parent directory is valid, but is an orphan, too. - The default date format is now FORMAT_DOS, which seems to work much better than FORMAT_DEF for date parsing. - Replaced the date+time spec processing code. DateToStr() was too clever by half and did not even flag an error if there was more text in the date/time spec than it could handle. This is why the date+time spec processing code now tries to parse things first before then letting DateToStr() take a crack at it. DiskDoctor 2.118 (20.8.2018) - Preparations for identifying directory entries whose parent directory or one of its parent directories is orphaned. This is a state called "out on a limb" which means that the directory entry is not reachable through the root directory, but it has a valid parent directory. Again, this state is transitive, just like the orphaned state. The plan is to avoid tagging complete unreachable branches as orphaned because we would lose the still valid directory structure which they are a part of. The local root of the disconnected branch can still be orphaned and will be tacked onto the global root directory. - Nameless directory entries are now consistently assigned names which feature the type of entry (e.g. "hard-link") followed by a "#" and the block number of the respective directory entry header block, e.g. "hard-link#12345". - The tests which determine whether a directory entry is an orphan now include an additional set of checks for unreadable blocks. - Updated the documentation again, so that it now covers all the command line options, including BLOCKSPERTRACK and FILEBUFFERSIZE. - Replaced all the (SetSignal(0,0) & SIGBREAKF_CTRL_C) tests with a signal exception handler that sets a global variable when the [Ctrl]+C keys are hit or a "Break" signal is sent. This should yield better performance and responsiveness since each signal check has to go through a Disable()/Enable(). Calling SetSignal() in a tight loop (as Disk Doctor does often) can cause pipeline stalls on the '040 and '060, so checking a global variable should be the better option. DiskDoctor 2.119 (20.8.2018) - The order of the parameters for the signal exception handler function matters for OS4, which requires that the SysBase pointer is passed as the first parameter, not as the last. - The signal exception handler must not make debug calls (too little stack space available). - Returned to opening the log file with MODE_READWRITE since ram-handler does not support ChangeMode() correctly. Oh well... - The user directory/file/hard link/soft link creation/last change date stamp could get lost in LNFS mode, which even filtered down to the contents of the DABI file. Practically every single file, etc. copied would feature an invalid date stamp. Fixed. - When copying file data, the log file no longer includes information on which data/extension block is going to be read next. This was generating far too much output with comparitively very little informative content. - The creation/last modification date stamp is no longer set for copied directory entries if it is completely invalid. DiskDoctor 2.120 (21.8.2018) - Aborting the copy operation could leave the current directory set to an invalid state. Fixed. - Copying less data than expected now correctly shows that the copy was truncated. - Simplified the file data copy operations, which no longer perform their tasks in two separate loops (first loop for the contents of the file header block, second loop only dealt with file extension blocks). This makes for shorter code and also provides an opportunity for another consistency test in case the file does not feature the expected number of extension blocks. - Error and progress messages now make more consistent use of the term "key" vs. "block". - Significantly improved file copy performance by figuring out how many data blocks may be read in sequence, and then actually reading and processing them in sequence. This is a lot faster than reading the individual data blocks one at a time, even for OFS data blocks which contain, among other things, checksum information. This still needs some more work. How many blocks may be read in sequence is limited by how large the read buffer is and how many data block references are stored in a file/file extension header block. This needs to be taken into account, as well as the buffer size used by the asynchronous file I/O functions which cannot make the best of the amount of data to be written if the buffer size is much, much smaller than the amount of data which has to be written. - The asynchronous file I/O functions now test for the Ctrl+C/Break status and will abort, if necessary. DiskDoctor 2.121 (22.8.2018) - Added memory allocation debugging code to surface any remaining resource usage issues. - Added crude resource tracking on top of the memory allocation tracking code. Don't do this at home, kids... - Preparations for managing the smaller memory fragments through memory pools. The upper threshold appears to be 8192 bytes. Most of the allocations (>70%) are smaller than this size. DiskDoctor 2.122 (22.8.2018) - Switched memory allocations for data which does not have to reside within a special type of memory over to use memory pools. This should help to keep memory fragmentation at bay and also run a bit faster since Forbid() need not be called for each single allocation. The new memory pool code is compatible with the resource tracking code, and also compatible with Kickstart 2.04 (V37). - When preparing for stack swapping, the new replacement stack is no longer allocated as MEMF_PUBLIC, which is in line with how CreateProc() and CreateTask() operate. DiskDoctor 2.123 (23.8.2018) - Verified that the new copying code works robustly with the SLOW option. - Removed obsolete code. DiskDoctor 2.124 (25.8.2018) - The log file now includes a bit more information about the device I/O configuration that is being used. Also, additional information is logged for some of the more unspecific errors flagged for inconsistent partition parameters. - The device I/O requests need to be allocated with the MEMF_PUBLIC requirement enabled. - Releasing the SCSI I/O command buffers was using the wrong size parameter. Ouch :-( - The default size of the disk read/write buffers is now limited to be no larger than about 72 blocks and no more than a total of 40 KB. The user can still override this with the BLOCKSPERTRACK option, though. The rationale behind the limitation is to conserve memory, such as for partitions which use several hundred thousand bytes per track, and to limit the negative performance impact of overly large data transfers. The performance gain afforded by large read buffers all too quickly turns into an impediment... - The device I/O buffer is now allocated with exactly the type of memory rquested and no longer includes the MEMF_PUBLIC requirement by default. DiskDoctor 2.125 (26.8.2018) - Simplified and unified the short/long DateStamp conversion. - Added macros to make the tests for fib_DirEntryType more readable. - Which disk command set (standard, TD64, NSD or direct SCSI access) is used is now logged correctly. The log also includes more detailed information on why a certain command set was used, and if the choice is unsuitable. - The fix_date_stamp() function did not correctly normalize the number of minutes. Fixed. - examine_all_blocks() now prints a warning for every single invalid date stamp it finds. - What was previously known as the "last volume modification date stamp" is actually the date stamp of the last change to the validation state of the volume, either before or after the "bitmap is valid" flag of the root directory was changed. It now is reported as such. Note that this date stamp did not exist in the original FastFileSystem (1987) and was introduced in 1990 (according to the April 1990 beta #5 FFS source code). DiskDoctor 2.126 (30.8.2018) - The hard link check now clarifies that linking to the root directory is not permitted. This limitation is in place because other than user directories, the root directory lacks a pointer to the list of links which reference it. - Shortened the progress information for the examine and copy operations a bit, to improve clarity and remove redundancies. - The Amiga file system considers a character unprintable if it is a control character (ASCII codes 0..31, which includes the tabulator, and ISO codes 128..159). Funny enough, the delete character (ASCII 127) is not considered unprintable, while is actually is... This is not a big deal, but it had to researched and clarified because the rules for displaying such file or volume names are different from the needs of the file system. - The repair mode is now a compile time option, which means that you would need to uncomment the '#define REPAIR_SUPPORTED' line in the "repair_volume.h" file in order to enable it. DON'T DO THAT UNTIL THE REPAIR MODE IS ACTUALLY TESTABLE! This change was necessary because it's altogether too easy to build the "repair_volume.o" file and link it against the program. Also, the command line template and error messages printed by Disk Doctor would make references to the repair mode which was almost guaranteed to make people curious about a missing feature... - Rewrote the part of the block scan function which looks into the metadata blocks to make better use of the fact that file, file extension, directory, hard link and soft link blocks share the same structure. Blocks of hard links to files and hard links to directories are structurally identical, which also allowed the size of the implementation to shrink further... - More detailed error and warning messages are printed by the block scanning operation, which now includes showing the names of the directory entries in question. DiskDoctor 2.127 (30.8.2018) - The block scanning code does not need to call AllocMem()/FreeMem(). Fixed. - When scanning a volume which is supposed to use only long name directory block format standard format (31 characters maximum) blocks can turn up. In many cases this may cause Disk Doctor to report that the block name is invalid and it also checks if this might be a standard format block. If the block appears to have a valid 31 character name, this name is now printed along with the error message. - Directory entry names containing reserved characters were not properly flagged as being invalid. Fixed. - The volume validation date/time is now reported as such rather than the incorrect "volume last change date/time". - The block scanner can no longer "promote" a block which contains no recognizable data to block type "unknown". Only unreadable/damaged blocks may be tagged with this type. DiskDoctor 2.128 (30.8.2018) - Repaired the OFS/FFS/DCFS name output, which was broken down into several lines of text prefixed by "Note: ". DiskDoctor 2.129 (31.8.2018) - The volume name was only printed if it was completely sound and free of invalid or reserved characters. Fixed. - Skipping unreadable or damaged bitmap blocks would have the side-effect of failing to keep track of which bitmap block exactly was being read and whose contents would be stored in memory. This in turn could lead to the bitmap processing function to read too many blocks. Fixed. DiskDoctor 2.130 (1.9.2018) - The bitmap and bitmap extension blocks are now registered and counted even if the block contents are damaged or entirely unreadable. - The maximum amount of memory allocated at a time (an estimate if memory pools are used) is now printed at all times again. - Unreadable blocks are now counted correctly. - Blocks known to be unreadable are now consistently tagged as being damaged, too, which simplifies testing for damaged data structures. - The log file now says "stopped" rather than "finished" if the user aborted the examine/copy operation. DiskDoctor 2.131 (2.9.2018) - Bitmap blocks are no longer misidentified as being unknown during block scanning if the root directory check already tagged them as being bitmap blocks. - If the file system signature is changed due to user request, this is now printed and logged. - If the file system signature found on the first block of the file system is unsupported, we now stick with the file system type specified in the mount data. This is printed and logged. - If the file system signature found on the first block is supported and used instead of what the default mount information said, this change is now printed and logged, too. - If the name of the file system device to examine/copy/repair does not match exactly what the user asked for, this is now printed and logged. - Fixed a maximum length test for file/directory/link names for OFS/FFS/DCFS. The maximum for these are 30 characters, not 31. For good measure, checked the use of all constants in the range 30, 31, 107 and 108, and replaced them with preprocessor constant definitions. DiskDoctor 2.132 (4.9.2018) - The time stamp of the last bitmap change has been present since the original BCPL file system (fh-iii) implementation and appears to have been added to the FFS in 1990 because the original FFS lacked it. Disk Doctor now always displays and checks this date stamp. - The progress message stating that DABI files are being read or written now includes a hint that this process may take a while to complete. DiskDoctor 2.133 (8.9.2018) - The memory tracking-enabled code for FreeVec() would call FreeMem(). Ouch :-( - Memory allocation sizes are now rounded up to the minimum fragment size in the memory tracking-enabled AllocMem() and AllocVec() code. - Disabled the remaining fragments of the copy debugging code which should not have been left enabled. They clutter up the debug output and are only useful for internal testing. The code can be reenabled through a preprocessor definition (commented out by default). - The default build mode now disables all debugging functionality. - Rewrote the error messages printed for block numbers which are found in directory hash tables that do not reference known directory entries. Previously, it was hard to tell what the problem was, exactly. - Added the NOREQ switch which keeps AmigaDOS error requester windows from opening. Disk Doctor will still detect and report these errors, it just won't get stuck. This should be useful for unattended copy operations and such. - Updated the error messages printed for files which claim to be larger than they can be. - Disk Doctor now reports directory entries named "*" (one asterisk character) as problematic because opening or attempting to delete these can have unpredictable side-effects. Files by this name will not be copied. - The copy operation wound up copying all orphaned files by default. Copying orphaned files should never happen unless specifically requested through the ALL or ORPHANED switches. Fixed. DiskDoctor 2.134 (9.9.2018) - When reading file data from an OFS volume, the type of each block presumed to be a file data block is now checked. If the type does not match, a warning message will be printed and the respective block will be skipped. - You can now control whether or not file data blocks which fail the checksum validation will be copied or not. Use the SKIPDAMAGEDDATA switch to skip these blocks. The default is to copy these blocks anyway (if the DAMAGED switch is in effect). Note that only OFS format volumes even have data block checksums which can be verified. - We no longer use AllocVec() or FreeVec(). The last holdout in disk_access.c now tracks the size of the allocation separately. DiskDoctor 2.135 (7.10.2018) - CreateDir() can return ERROR_OBJECT_IN_USE or ERROR_OBJECT_EXISTS if the directory already exists, and ERROR_INVALID_COMPONENT_NAME if the destination is the root directory of a volume. A shared lock on the object, such as may be present because a program uses it as its current directory, may prompt the ERROR_OBJECT_IN_USE code. This is what prevented the COPY operation from working on the FFS. It did work with RAM:, though, and the FFS 'C' reimplementation. - Disk Doctor did not release the shared lock on the destination directory prior to beginning the copy operation. This contributed to the copy operation failing. DiskDoctor 2.136 (14.10.2018) - Replaced the code which created directories by a common solution which is a bit easier to get a handle on than the previous solution which tried to create the directory first and then sorted through the various error codes produced if the creation failed. The ambiguity of the error codes returned by various different file systems made a more complex approach necessary. The new code works mostly like the directory creation process of the "Copy" shell command. First it checks if the directory exists and is in fact a directory or volume. If the directory does not exist (IoErr() returns ERROR_OBJECT_NOT_FOUND) an attempt will be made to create it. If this fails, the error code returned will be examined and another attempt will follow to obtain a shared lock on the directory just like the previous code did. DiskDoctor 2.137 (23.11.2018) - The function which determined the available stack size, prior to allocating a new stack of sufficient size, did not return the proper information for a shell program (which Disk Doctor happens to be). DiskDoctor 2.138 (25.11.2018) - The I/O buffer size limitation which was intended to conserve memory for disk partitions with very large track sizes backfired with block sizes > 20 KB. The new code now makes sure that the buffer size is a multiple of the block size and will limit itself to use a maximum of two blocks worth of memory only. - Stack size threshold testing now takes into account that the original CLI stack may have already been replaced. DiskDoctor 2.139 (24.12.2018) - Replaced the algorithm which calculates the expected time remaining until the block scan operation will be finished with something a little more sophisticated. The new code calculates the number of sectors processed per second whenever another read request has returned. In order to constrain the volatility of this figure, an exponentially weighted moving average is calculated, which is then used for displaying the expected remaining time. - Shortened the progress report text printed during the block scan operation, omitting the sector number. What remains is the key, which stands for the "number of blocks" read so far. DiskDoctor 2.140 (25.12.2018) - Made the calculation of the estimate of the time remaining for the block scanning operation more robust. At the very least we need to make sure that the results of the calculations are valid real numbers. DiskDoctor 2.141 (3.1.2019) - Added formatted output sanity checking to the GCC 68k test build. This helped to detect invalid parameter usage in several places throughout the command. Not all of these output errors were entirely harmless :-( DiskDoctor 2.142 (3.1.2019) - Found more formatted output errors in the debug code (oh, the irony...). DiskDoctor 2.143 (4.1.2019) - Added the PRIORITY option which can be used to change the task priority which Disk Doctor will use while it is running. The original CLI task priority will be restored before Disk Doctor exits. The priority should be in the range -128..127 and will be truncated if it exceeds these limits. - The amount of data stored in the files on the volume is now counted and printed/logged. These are actually two figures, the first being the total sum of the file sizes as stored in the file block headers, and the second the number of data bytes which Disk Doctor may be able to read from these files (which excludes the contents of unreadable/corrupted data blocks). These figures are printed as the number of respective bytes, as well in GiB/MiB/KiB. If there are differences between these figures, a message will state how much data is missing or unexpectedly available. DiskDoctor 2.144 (12.1.2019) - Moved the calculation for the time remaining to complete the block scan operation into a separate file. The copy operation needs the same kind of prediction functionality. - Added support code which allows the EClock timer value to be used as a monotonic source of time passed since a reference point. This should provide for a more reliable foundation of time measurement than the timer.device GetSysTime() function, whose result is subject to the system time being changed. Mind you, this code still needs to be plugged in... - Whenever the file examination discovers something to be amiss with the file data blocks, such as these being used more than once or refer to a metadata rather than data, Disk Doctor now tries to resolve the full path name of the directory entry in question, for reference. - Bumped the number of I/O requests which will be used for block scanning to three (used to be two). The third one is necessary for the new function which resolves a block number to a full directory path (if possible), so as not to interfere with regular directory data structure processing. It also seems to speed up the scanning process :-) - The initial block scan now tries to identify metadata blocks which do not fit the standard/long name mode format, identifying them if possible. This avoids triggering an avalanche of warning messages for orphaned/deleted directory entries which so happen to be in long name format when the file system is expected to be in standard format, and the other way round. DiskDoctor 2.145 (13.1.2019) - When checking the consistency of a file and encountering block numbers which do not belong into this specific file, or which are entirely the wrong type, both the file being examined as well as the directory entry which somehow overlaps with the file are being tagged as damaged. This is necessary because Disk Doctor cannot tell which of these directory entries is the "true owner" of the blocks. Only the use may be able to tell which file contents are correct and which are not. - Plugged in the EClock reference timer code. DiskDoctor 2.146 (20.1.2019) - Simplified the calculation of the number of file extension blocks expected. This takes into account that the file header might report fewer than the maximum number of blocks expected (unlikely, but we have to follow the rules). - Replaced memcpy() with memmove() for consistency within the code base. - Removed duplicate and triplicate code from the file examination operation. Just for once Disk Doctor became shorter as a result. - Updated the error and warning messages to provide more detailed information. - Removed error and warning message information which was repeated several times over with the same content. Once should be sufficient. - Error messages for data in the wrong file, metadata where data was supposed to be, and for multiple use of the same block now include both the path of the file and and whaever the other guy is, just so you can compare both more easily. DiskDoctor 2.147 (20.1.2019) - Added additional diagnostic code which is only part of the debug build in support of the problem with data in the wrong file, metadata where data was supposed to be, and for multiple use of the same block. If such a problem is detected, this support code will go about reconstructing the list of all the blocks associated with the files/directory entries in question. It then verifies if the block which caused the problem is indeed present in both directory entries and then proceeds to print a list of all the blocks shared between the entries. So far testing has shown that if the defects are detected, then the new support code confirms the presence of the defects. DiskDoctor 2.148 (25.1.2019) - A comment block set for a directory is now tagged as being associated with that directory (as its parent). - All the directory entries stored in a directory are now tagged as being associated with that directory (as its parent). - The duty to mark file system blocks as allocated now rests solely with the functions which check the directory structures. - If the file examination finds metadata, it now reports the file header key number of the directory entry, etc. in question, and not just the path name and block number. - The root directory block is no longer tagged as being in use by examine_root_directory(). The job now solely rests with examine_directory_contents(). - Fixed the progress report output which should not have used Printf(). - Added fixes/casts to cope with compiler warnings. - Finally figured out why get_path_name_for_block() had to read each metadata block from the volume rather than check the in-memory file system tree instead. The file data flagged as metadata blocks did not appear in the in-memory file system tree, which it should have. This should never have been the case. The root cause of this problem was in tagging data blocks as "reachable", which allowed orphaned/deleted metadata blocks to be misidentified as being valid (and corrupted) files, etc. DiskDoctor 2.148 (25.1.2019) - When examining a file data block, checking the block's contents as retrieved from the volume must involve verifying that the type information in that data matches what was expected. This check was missing. - The checksum and type of the block data retrieved for file extension blocks were not checked properly. Fixed. - The order in which the file examination performed the checksum verification and the verification of the block contents for file data was reversed. The checksum verification should have always come first. - When "promoting" block to be an FFS data block both the number of data blocks found and the number of blocks of the type which the block was previously considered to be one of are now adjusted correctly. - The report of the amount of file data reported and recoverable was misleading since it included both the files in good condition and the orphaned/deleted files. These values are now counted and reported separately. If both figures are identical only the first is printed, otherwise both figures will be shown with an explanation of whether there was more or less data found than expected. DiskDoctor 2.149 (28.1.2019) - If Disk Doctor finds directory entries which may be in a different format than what it was currently looking for (e.g. OFS/FFS/DCFS directory entries on an LNFS volume) it will count how many it found and later (in verbose mode) hint that it may be worth repeating the examination/copy operation by using a different file system flavour. - Suggesting that the user repeats the scan/copy/recovery operation for long name mode/non-long name mode files requires that there are options for turning the long name mode off and on as needed. I have added these options as well as those which turn off the fastdir and international modes, and changed how the existing FFS, OFS, LNFS, FASTDIR and INTERNATIONAL options work. If the volume already uses a well-known file system signature (ID_DOS_DISK, ID_FFS_DISK, ID_INTER_DOS_DISK, ID_INTER_FFS_DISK, ID_FASTDIR_DOS_DISK, ID_FASTDIR_FFS_DISK, ID_LONGNAME_DOS_DISK or ID_LONGNAME_FFS_DISK) then OFS, FFS, INTERNATIONAL/NOINTERNATIONAL, LONGNAME/NOLONGNAME and FASTDIR/NOFASTDIR options will modify the signature according to what the user requested. If the volume's file system signature is not well-known, Disk Doctor will construct a completely new signature based upon what the OFS, FFS, INTERNATIONAL/NOINTERNATIONAL, LONGNAME/NOLONGNAME and FASTDIR/NOFASTDIR options say. Note: The Disk Doctor documentation still needs updating... - The hint that the volume contains minor defects which could be repaired did not print the correct text. Fixed. - Improved the robustness of the block scanner with regard to identifying named directory entry blocks which may not be in the format it expected, such as orphaned/deleted FFS files on a long name mode (LNFS) volume. The code now first checks if there is something not quite right with the name or comment and only flags the block for a closer look if there seems to be a good reason for it. DiskDoctor 2.150 (31.1.2019) - If the number of data blocks referenced by a file header or file extension happens to be negative (huh?) we now print a warning and assume that there are 0 blocks referenced. - The user can now override the block size of a volume via the new BLOCKSIZE option. Note that the sector size cannot be changed, only how many sectors make a block. The block size must be given as the number of bytes to use, e.g. 1024, 2048, etc. rather than 2, 4, etc. for 512 bytes per sector. Disk Doctor will try to find a suitable number of sectors per block and print an error message if it fails to come up with one which satisfies what the user asked for. If the block size is changed, the verbose output and the log file will include a note to this effect. Note: The Disk Doctor documentation still needs updating... - The user can now directly override the file system signature of the volume with the new DOSTYPE option, e.g. DOSTYPE=0x444F5301 instead of using a combination of the INTL/NOINTL/LNFS/NOLNFS/FASTDIR/NOFASTDIR modification options. As with the block size, the verbose output and the log file will include a note detailing if the file system signature was overridden. DiskDoctor 2.151 (31.1.2019) - Changed the order of the command line options which control the file system type and the block size. - The file system type expected by the DOSTYPE parameter no longer needs to be an ASCII hexadecimal number. It can be one of "DOS0" through "DOS7", too, which corresponds to how the "Mount" command's "DOSTYPE" documentation covers the purpose and effect of this option. - Updated the documentation to include the INTL/NOINTL, LNFS/NOLNFS, FASTDIR/NOFASTDIR and DOSTYPE options. DiskDoctor 2.152 (1.2.2019) - Removed a duplicate error message which was printed if the block size the user asked for was unsuitable. - If the name of a directory entry cannot be reconstructed by examine_all_files() it's no longer grounds for stopping the examination. We just won't be able to show the name, but the keys could still be useful. - As a precaution, the code which tries to reconstruct the path of a directory entry now makes sure that it won't get stuck in a circular reference loop. DiskDoctor 2.153 (3.2.2019) - Removed the LNFS name access hack from examine_all_blocks.c and replaced it with a more robust solution. - Removed duplicate code from copy_directory_tree.c in order to simplify the operations. The new code checks for [Ctrl]+C more frequently, and it watches more carefully how much data it was supposed to copy. - Added the SPACENEEDED option which can be used as a safeguard to prevent Disk Doctor from running out of space during the copy operation. - Disk Doctor now counts how many file data blocks are associated with the files it finds and tries to figure out if these blocks are in OFS format, or maybe in FFS format. This information is used later to hint that it might be useful to repeat the examination/copy operation with the respective OFS or FFS switch enabled. - Updated the documentation. DiskDoctor 2.154 (10.2.2019) - Turns out that TD_READ64 commands can crash AmigaOS4 hard disk drivers. Who knew... This is why the AmigaOS4 version now defaults to use the NSD read, etc. commands, whereas the AmigaOS 2.x/3.x compatible version sticks with TD_READ64. - Adapted the various varargs functions to work well with AmigaOS4. This finally produced the first working AmigaOS4 port of Disk Doctor. - The low-level disk I/O command set (TD64, NSD TD64, automatic) did not exactly follow the user's choices, or at least the code was not clear about how it was honoured. I have rewritten it to do better in this respect. DiskDoctor 2.155 (10.2.2019) - Because an additional I/O request is needed for the debug build, the "disk_access.h" header file has to be able to know whether the debug build is called for, just to be on the safe side. - Turns out that the AmigaOS 68k method for avoiding frequent calls to SetSignal() just for the purpose of [Ctrl+C]-checking is not stable on AmigaOS4. So we go back to calling SetSignal() again just for the AmigaOS4 port... DiskDoctor 2.156 (18.2.2019) - Switched from built-in SAS/C IEEE math library to mathffp.library, which appears to be just about sufficient for our purposes, although the reduced accuracy can cause the progress display to temporarily stop showing a useful estimate. Note that we use mathffp.library because it is in ROM and mathieeesingbas.library cannot be used through the SAS/C math libraries. - Disabled the Kickstart 1.x compatible shell stack space check which seems to be obsolete, wrong or maybe both. We now trust the Task stack boundaries to be sufficient and correct. - Enabled the SAS/C "utillib" code generation option which also helped to reduce the program size. Thank you, Thomas! DiskDoctor 2.157 (21.2.2019) - Added options for building a Disk Doctor version which requires Kickstart V39 or better. The default is to require Kickstart V37. Dropping V37 compatibility makes Disk Doctor shorter by some 470 bytes. - Added options for building a Disk Doctor version without support for asynchronous file I/O. Omitting this feature will result in reduced performance on Kickstart 2.x/3.x when recovering data from a volume. It does save about 2000 bytes of code, though. - Dropping the newly-introduced utillib build option saves about 170 bytes of code. Who knew? - Failing to open mathffp.library, if required, now prints an error message to this effect. DiskDoctor 2.158 (20.3.2019) - The expected time to complete the block scan operation is now optional if the required math support library cannot be opened. In this case no projected completion information will be calculated or shown. Disk Doctor is supposed to be a fully self-contained tool which should not depend upon specific libraries, etc. to be on the disk it was started from. The current AmigaOS 3.1.4/3.2 build does not necessarily contain any math#?.library and Disk Doctor must be able to cope with this restriction. - Just to see if it has a positive effect, the projected completion time of the block scan operation is now calculated using the single precision IEEE floating point library "mathieeesingbas.library" rather than "mathffp.library". With the algorithm used to calculate the completion time IEEE floating point operations are likely to generate more robust results than with the FFP format. DiskDoctor 2.159 (26.6.2019) - Reworked the directory tree management code to yield proper error codes where previously only a pointer (which could be NULL to indicate an error) would provide any clues as to what might have gone wrong. - Edited the built-in error message description text to add more detail, which should help to wring some more useful information out of the errors which may occur when reading DABI files. - Removed "simple_list.o" from the build list since it is needed only by the currently disabled "repair_volume.o". Disk Doctor might actually become a little bit shorter as a result. DiskDoctor 2.160 (27.6.2019) - Added diagnostic messages to the DABI processing function which should explain in a little more detail what went wrong in case the contents of the DABI file were found to be corrupted or incomplete. - Replaced the error code values for certain functions so that the problem they need to report is represented better with less ambiguity. DiskDoctor 2.161 (28.6.2019) - Turns out that there is a lot of zlib code which is not needed by Disk Doctor and which can be elegantly removed whilst at the same time saving some 4500 bytes of code. How about that? - Added more diagnostic code to "dabi.c". Which block information is written and read, as well as which directory information is is read, is now sent to the debug log. - Looks like the source of all the known DABI file troubles was a wrong call to set_tree_item_jmp() in "examine_all_files.c" line 2212 which had the parameters switched. It should have recorded a file header block being the parent data structure for a file data block, but it ended up recording the file data block as the parent data structure of the file header block. This error had profound side-effects on Disk Doctor's ability to verify the consistency of the file system. It would mistake a perfectly sound file system structure for being corrupted. The bug has been around since at least version 2.2, going back to October 2017... Thanks go to Thomas Klein and Mikhail Malyshev for providing and verifying the test data which allowed this bug to finally come to the surface! ----------------------- AmigaOS 3.2 project -------------------------- DiskDoctor 2.162 (24.11.2019) - Saved a couple of bytes by zero-initializing the BSS segment. DiskDoctor 2.163 (29.11.2019) - Replaced the compiler-built in setjmp/longjmp code by assembly functions to avoid some code overhead. DiskDoctor 2.164 (2.12.2019) - Plugged in an iterative merge sort function for singly-linked lists which promises to save size by eliminating duplicate code and which goes easy on the stack size usage. The sort function was adapted from Simon Tatham's "Sorting a linked list using Mergesort" example code which can be found at https://www.chiark.greenend.org.uk/~sgtatham/algorithms/listsort.html DiskDoctor 2.165 (2.12.2019) - Let's see if we can shrink this giant program further by some hundred bytes through use of the utility.library math functions. - A single structure copy can be replaced by CopyMem() if we want it to be replaced, and save another hundred bytes along the way. - Replaced the qsort() function. It's not shorter than the SAS/C runtime library version, but it promises to use less stack to do its job. Replacing the local swmem() function with the SAS/C runtime library code could save another 30 bytes. DiskDoctor 2.166 (2.12.2019) - The directory tree entries are sorted both by name and by block, through different list members. The new list sort code did not account for this difference. Fixed. DiskDoctor 2.167 (2.1.2020) - Happy new year! It turns out that support for 5.25" disk drives could suffer because of a missing flag in the parameters used for opening "trackdisk.device" units 0 through 3. The disk access code now adds this flag, if necessary. DiskDoctor 4.168 (14.5.2020) - Rebuilt with optimization for time enabled. Previously, optimizations for both time and size were enabled, with consequences hard to fathom... DiskDoctor 2.169 (21.5.2020) - The local zlib was built with both optimizations for time and space enabled as well. Rebuilt it with optimizations for time enabled only. DiskDoctor 2.170 (17.6.2020) - Now accepts "trackfile.device" as sufficiently safe not to crash instantly when called upon to process the TD_GETGEOMETRY command. Also, the device name comparison is now case-sensitive since OpenDevice() uses case-sensitive name comparison, too. DiskDoctor 2.171 (2.8.2020) - Hooked up the raw floppy disk track data decoding functionality with the disk access code. It's not yet operational because the command line switch that would activate it is not yet available. The read error handling/retry mechanism is not implemented yet and neither is the synchronous block read operation. - Removed the unused merge sort code. DiskDoctor 2.172 (17.8.2020) - Better integration of the raw disk access code for the scanning stage of the recovery process. While it is limited to read entire tracks and then, if necessary, breaking them down into the sectors needed, the interface code in disk_access.c will prepare the data to look like what a normal CMD_READ would have produced. If the sectors required are not all present because of decoding errors, multiple attempts will be made to reread the track on which they reside, collecting the best "versions" of the sectors. How many retries may be made depends upon the global retry limit, which can be either a maximum number > 0 or 0, the latter meaning that an unlimited number of retries will be made until either everything reads correctly or the user has decided to skip to the next track by hitting [Ctrl+E]. Both the use of the raw read option and the retry limit need to be hooked up to the ReadArgs() template, etc. - Raw disk access now insists on the size of a block to be identical to the size of a single sector. DiskDoctor 2.173 (18.8.2020) - When using raw floppy disk access, the copying operation will never read more than one data block at a time. This allows for better control over the read retry operations which the user should be able to skip, if necessary. - The single block read operation now supports retrying after fewer sectors were read from a floppy disk than expected. It currently does not show that reading is being retried, though, but the retries can be skipped and the read operation can be aborted, too. - The number of missing sectors is now displayed correctly if raw floppy read access failed to read a full track. - The number of retries for raw floppy read attempts now defaults to 10. The minimum is 0, which means that Disk Doctor will make one single attempt to correctly read a track and then move on. - Hooked up the code which enables the use of raw floppy read access. Time to finally test it :-) DiskDoctor 2.174 (19.8.2020) - Added more debug output to the new raw floppy disk access code. - Rewrote the code used by the file copying operation to make better use of table which contains consecutive sequences of data blocks which may be read more quickly in one piece rather than individually. It should also be a bit easier to understand how the code works ;-) DiskDoctor 2.175 (19.8.2020) - Raw floppy disk access finally seems to work for the happy path, namely examining and copying data without retrying track read operations. Damage simulation still has to be implemented for the raw read commands. - Stopping the block scanning operation for the entire disk could run afoul of code which I had left in the cleanup code by mistake. Fixed. DiskDoctor 2.176 (22.8.2020) - Added raw floppy disk damage simulation. Raw sector data damage is simulated, in contrast to simulating damage to the decoded data. Every damaged sector will read back incorrectly for up to 6 read retries after which the damage will go away. This is supposed to simulate the sector decoding operation succeeding in producing the correct bit pattern after all. - The damage simulation feature is now only built if Disk Doctor is also built with the debug output enabled, too. This is to make sure that the production version has neither debug mode nor damage simulation enabled. - Removed the unused old mergesort code for good. DiskDoctor 2.177 (22.8.2020) - Updated the documentation which introduces the new options for recovering files from floppy disk media. - The damage simulation for floppy disk data did not modify the encoded raw data correctly. Fixed. DiskDoctor 2.178 (2.12.2020) - If a volume's root directory block contents state that the volume is not currently validated, Disk Doctor will report it now. Previously, only defects in the volume bitmap would produce such a note. - Removed block contents validation and reporting of issues for individual blocks from the block scanner. This never made great sense because most of the same validation would happen later anyway when the file system structures were examined. The changes made now avoid reporting defects for block contents during the initial scanning stage. These will be reported, if necessary, during the later stages. One reason why reporting any defects during the early block scanning stage was not helpful was in that it happened both for blocks which would later turn out to be part of the file system data structures, as well as those which would not feature in it. The latter could happen if you deleted files or reformatted a volume using a different file system mode. So you would have seen warnings, etc. for metadata and data blocks which did not necessarily matter. - For every defect detected in the file system data structures Disk Doctor will now print an error message (if enabled). Previously, some of these defects would be detected and blocks would get marked as being defective, but there was not necessarily an explanatory message to go along with these. - Defect reporting now covers not just directories, files and links but also the data structures associated with them. This matters mostly in the Disk Doctor reporting stage after the volume has been examined. - Removed a bit of unused code... DiskDoctor 2.179 (3.12.2020) - Reviewed the code which tags blocks as being "reachable" and removed those instances which make no good sense. The "reachable" attribute is for metadata and not so much for data blocks, comment blocks (LNFS only) or bitmap blocks. The "reachable" attribute is relevant for reporting serious defects, which is why defects for individual bitmap blocks and bitmap extension blocks are now recorded for the root directory (which the bitmap is associated with). DiskDoctor 2.180 (5.3.2021) - When examining all blocks of the volume (reserved blocks excluded, though), special measures are taken for partitions/disks whose parameters match what a high/double density 40 or 80 track floppy disk would feature. For such volumes the read operations are now aligned with the boundaries of the individual tracks. This makes it easier to interpret read operations which fail with an error code set. If the error code came through a checksum or physical error on the medium, which for trackdisk.device condemns the entire track to be unusable, this now only affects this particular track. Previously, such errors would have resulted in marking blocks in neighbouring tracks as unusable, too, which would not have featured in the failed read attempt at all. DiskDoctor 2.181 (6.3.2021) - Verified that the changes which now make sure that for floppy disk drives only full tracks are being read do work out well. As a side-effect, this rendered some of the sanity checks in the code redundant, resulting in modest code size savings. - Updated the warning message for the validation record which stated major differences without stating what the difference would refer to in context. - Updated the warning message (in verbose mode) which states how many invalid entries there are in the block allocation map so that it covers both the singular ("entry") and plural cases ("entries") correctly. - The query data buffer to be filled in by the NSD query command, when talking to the disk driver, is now zeroed before it is sent to the driver. Just in case it might make a difference. - Added that extra bit of semantic sugar to the many block number variables which are now of type "block_number_t" when they previously were of type "ULONG". DiskDoctor 2.182 (20.3.2021) - As a workaround for floppy disk error handling which seems to lead to Disk Doctor hanging under certain unclear conditions, the built-in raw disk access mode is now enabled by default in auto-detect mode. If the disk/partition matches a 5.25" or 3.5" 40 or 80 track Amiga floppy disk and the name of the disk driver happens to "trackdisk.device", Disk Doctor now automatically enables raw disk read mode. You can override this through the RAWFLOPPYACCESS option which can be one of AUTO, ALWAYS or NEVER. The documentation has been updated to reflect this change, with the RAWFLOPPYACCESS switch now requiring a keyword. DiskDoctor 2.183 (21.3.2021) - Modified the processing of the RAWFLOPPYACCESS option so that the choice of NEVER will not cause the DISKACCESS option to be ignored. This is a fix for a bug which I introduced :-( DiskDoctor 2.184 (9.4.2021) - The arguments for the RAWFLOPPYACCESS option were checked in the worng option (DISKACCESS), which had the effect of rendering the RAWFLOPPYACCESS largely useless :-( Thanks go, as so often, to Brian Carpignano who found the issue in the nick of time :-)