locale 44.1 (5.8.99) - Fixed an ancient buffer overrun problem in the DateToStr() patch. This was most apparent with country definitions that had four digit year numbers in them, such as italia.country or portugal.country. Since the length of a day/date/time string, as passed to DateToStr() cannot be greater than 16 characters (including the terminating NUL), a date string such as generated for the month December (e.g. " 1-Dicembre-1999", which is exactly 17 characters in size, including the terminating NUL) would cause the buffer to overflow. This caused all kinds of trouble. I limited the size of the string generated. While this is not exactly a good fix for the problem (the last year digit will "drop off"), it avoids the buffer overrun. - Added default cases for the RawDoFmt() patch (no longer throws Enforcer hits on invalid formatting commands) and the ARexx character conversion code. locale 44.2 (5.8.99) - Changed the built-in default locale for united_states to use four digit year numbers. locale 44.3 (9.9.99) - In DateToStr() the '%y' code would calculate the year number minus 1900 to obtain the two digit year number. I changed this to use the year number mod 100, which should do it. locale 44.4 (24.9.99) - The library now maintains proper usage counters for all locales it loads. Previously, locale.library would keep any locale loaded in memory indefinitely, including the associated language driver. Now everything that remains is the sys/dos.catalog file which, since dos.library does not and cannot maintain a usage counter for its strings, has to remain in memory once loaded (some dos.library users may cache strings). - The library no longer bumps its usage counter every time the default locale is changed. This happens only once now, when the operating system patches are planted. - The built-in "english" locale now has its usage count set permanently to 1, preventing it from ever getting removed. locale 44.5 (24.9.99) - The ParseDate() code now specifically allows "%y" year numbers to have more than two digits. locale 44.6 (7.11.1999) - The catalog file loader no longer refuses reading catalog version strings if they are longer than 99 characters. It now reads a maximum of 99 characters and tries to make the best of it. - When planting its patches with SetFunction(), locale.library now brackets all calls with Forbid()..Permit(). locale 46.1 (12.3.2002) - Replaced 32 with MAXLANGSIZE or MAXLOCSIZE where appropriate. - Added name length checks. - Renamed charset-related internal functions from "English" to "Latin1". - If a language driver provides a GetCodeSet function and this function returns a non-zero codeset and the language driver does not provide all charset-related functions, a charset driver is loaded (e.g. Locale:CharSets/111.charset for ISO 8859-15 Latin9, the number is the IANA charset number, see http://www.iana.org/assignments/character-sets) and used for the charset-related functions not provided by the language driver if it provides replacement functions for the builtin Latin1 functions. - Now tries to open diskfont.library V46+ (no complain if it fails) and sets diskfont.library V46+ default charset from default locale loc_CodeSet. - Created 111.charset ISO-8859-15 Latin9 charset driver. - Created ISO15 language drivers from the available sources (only had the OS3.1 language driver sources available). No danskISO15.language driver, sorry, the dansk.language driver has some strange builtin charset function replacements I dont understand yet... locale 46.2 (20.3.2002) - Now needs diskfont.library V46+. - Now uses the MIME name instead of the IANA number for the charset driver in Locale:CharSets/, e.g. Locale:CharSets/ISO-8859-15.charset. - Renamed 111.charset to ISO-8859-15.charset. - Renamed ISO15 language drivers to ISO-8859-15. locale 46.3 (22.3.2002) - Translated latin1.asm to ISO-8859-1.c - Fixed Collate1 table bugs from original "english.asm" driver. - Fixed CharAttr table bug from original "english.asm" driver. - Fixed alien memory reads and random results of IsXXXX functions and ConvToUpper/Lower functions for characters > 0xFF. locale 46.4 (25.3.2002) - Fixed ToUpper/ToLower/CharAttr table bugs for '�' and '�' from original "english.asm" driver. locale 46.5 (26.3.2002) - Added diskfont includes. locale 46.6 (26.3.2002) - Moved TA_CharSet to diskfont.h. locale 46.7 (23.7.2017) - For better compatibility with legacy workbench versions, this release no longer requires diskfont V46, but uses it if it is available. - Minor improvements, uses now utility for the math, all const data is now in the code segment, avoiding relocation. locale 46.8 (5.1.2018) - Disabled support for character sets right now as 3.1 does not have them. We may reconsider this feature later. locale 46.9 (19.8.2019) - Very minor optimizations in the locale.library Format() function. locale 46.10 (6.6.2020) - Modified RawDoFmt patch to be compatible with exec 45.x (putCharProc may be NULL) locale 46.11 (7.9.2020) - Reduced the number of header files included for prototype/interface functions. - Added wrappers for the various ASM, REG and __far keywords which match what's now in . - No functional changes... yet! locale 46.12 (9.9.2020) - Reset the code indentation, scrubbed the number of empty lines which separate functions and code, added a few more empty lines to make the code a bit more easy to read. - Still no functional changes... yet! locale 46.13 (9.9.2020) - Lucky #13! - Dialed back the exuberant code indentation, in particular for the nested switch/case statements. - Added { } braces to code for which it renders the results more readable. - Still no functional changes... yet! locale 47.1 (9.9.2020) - Rewrote the OpenCatalogA() function so that it minimizes the time it may hold onto the global LocaleBase lock. This no longer happens until a catalog matching the requirements is found to be already in memory, or until a catalog matching the specs has been read. In case it turns out that a different Process has already read that catalog first, then that will be used instead. - Trying to load a catalog from disk now only makes two attempts to open it, and not three. If the catalog can be found in PROGDIR:Catalogs, then that's fine. If not, it will check LOCALE:Catalogs instead, and if that doesn't work, then there will be no second attempt to maybe find it in PROGDIR:Catalogs after all. locale 47.2 (11.9.2020) - Replaced short integer type variables with normal int type variables. - OpenCatalog() no longer checks for every language if PROGDIR: is valid. Once is enough. - Replaced implicit non-zero value checks with proper explicit tests, using the correct type. - Replaced untyped 0 with the specific correct type, e.g. NULL or '\0'. - Where strings are compared, the result is no longer tested against 0, but against a preprocessor macro by the name of SAME, to make it easier to spot what is being done. - Broke up combined assignment + test (e.g. if(result = AllocVec(..))) into separate assignment and test where possible. - Hard-coded data structure and table sizes ("magic numbers"), e.g. 10 and 16, now use either a sizeof() of the type in question or a macro which produces the number of table entries. - When setting the global "language" environment variable, the name of that language is now truncated before it is copied into the name buffer passed to SetVar() instead of after it has been copied. This avoids trashing memory or the Process stack for language file names longer than 22 characters. - OpenLocale(), if called with a locale name, no longer hangs onto the global LocaleBase lock when loading the requested locale from disk. It's not necessary to do so. - Replaced Pascal/Modula-2 style while-loops with their equivalent 'C' for-loops for better readability. locale 47.3 (12.9.2020) - The loop tests in ISO-8859-1.c broke the library. Oops! Back to square one. We shamble along with changing the table index variables from bytes to integers. - OpenCatalogA() did not process all the languages in the table because the code which does that is at the very bottom of the loop, so you can't safely use "continue" anywhere. Bummer :-( - Figured out why InitNumFormat() only checks the first 10 entries of locale->el_Locale.loc_Grouping[]. The size comes from the locale preferences files, specifically the CountryPrefs section and its 'cp_Grouping' table. locale 47.4 (12.9.2020) - No longer assumes that the locale->loc_LanguageName will have a ".language" suffix. Since OpenLocale() does not require this, expecting that the last 9 characters of the locale name can be removed is not helpful. If the name happens to be shorter the stack would be trashed, or worse. A faulty locale prefs file could trigger this :-( - Checked which function parameters are better declared as being constant. Adapted the .sfd file and prototypes and interface files accordingly. - The number formatting helper table setup was subject to an off-by-one error and could have initialized a 17th entry out of 16... locale 47.5 (13.9.2020) - The string data which is part of a catalog file must contain at least one translated string to be useful. The code which preprocesses the data for the catalog string lookup function depends upon that. Up until now this requirement was never even tested. - Preprocessing of the catalog file's translated string data now verifies that the size of the string data is not larger than what the buffer allocated for it would hold. If the size of a string is found to be invalid, preprocessing will be stopped immediately. This at least provides a chance to keep any valid translation strings which were valid. - If the catalog string data is found to be unusable, the catalog data will be freed and the catalog rendered safe for the catalog lookup code. - If the catalog version string is malformed or corrupted, trying to process it is now aborted as soon as possible. locale 47.6 (14.9.2020) - Small changes for (hopefully) better readability. locale 47.7 (19.9.2020) - Replaced the assembly language version of the GetCatalogStr function with plain 'C' code. Not at straightforward as the original version but the lookup loop itself is not so different. locale 47.8 (19.9.2020) - Replaced the catalog string initialization with code that uses a data structure to show how the process works. Pointer and integer arithmetic will only get you so far. - Removed the assembly language GetCatalogStr implementation. locale 47.9 (19.9.2020) - Simplified both the LoadCatalog() and LoadLocale() implementations so that they no longer do things multiple times for cleanup work (in case of error). Also, the code indentation no longer grows with each if (..) clause, rendering the code a bit more readable. - If OpenLocale() failed to actually open the locale, having allocated memory for it first, it would have freed that memory twice. This appears to be a bug which goes back to at least 1992. Fixed. locale 47.10 (19.9.2020) - If there are at least 128 entries in a catalog file, locale library will now try to build a lookup table for the catalog strings, sorted by string ID at OpenCatalog() time. If enough memory is available, that is. GetCatalogStr() will then make use of this table to perform a binary search for the catalog strings. This should make lookups for large catalogs noticeably faster, but benchmarking is still needed... - Added a non-recursive local qsort() implementation in order to avoid pulling in the entire qsort() and stack management code for them SAS/C runtime library. locale 47.11 (20.9.2020) - Fix for free after free -- a classic copy & paste error. Thanks go to Thomas Klein for finding and reporting it. locale 47.12 (20.9.2020) - Changed the optimization options for the build so as not to inline local functions. This is to limit the amount of stack space used. - Moved the locale initialization out of the LoadLocale() function in order to simplify the code. locale 47.13 (20.9.2020) - If loading a catalog from disk fails because there is not enough memory available, we now check if the catalog has already been loaded (by a different client) before we give up. It might just work... - Moved three separate instances of the code which tries to reuse an existing, cached catalog into a single separate function which can be conveniently reused. - Freeing unused catalogs did not free the catalog entry search table. Fixed. locale 47.14 (21.9.2020) - Updated the code documentation. - Changed the names of certain function parameters as well as local variables so as to better convey their respective purposes. - CloseLocale() no longer decrements the usage counter of a locale if its counter is already 0. locale 47.15 (22.12.2020) - Added experimental replacement code for the FormatString() function which is hopefully more robust and also faster than the original implementation. This replacement is not enabled by default and requires recompilation of the "format.c" file with the preprocessor symbol "USE_NEW_FORMATSTRING" defined. locale 47.16 (23.12.2020) - Now stores the actual sizes of the parameters instead of 1 (short) and 2 (long) and later multiplying that by 2. This actually saved a few bytes of code. - Pointer type casting tricks made the different conversions for signed and unsigned 16 bit integer/character parameters unnecessary. - Updated the AutoDocs to mention that in all version including 47.14 the string produced by the %s conversion specifier was limited to a maximum of 65535 characters. This limitation has been removed. locale 47.17 (23.12.2020) - If possible, NUL-terminated strings are now copied to the output until the '\0' character is found rather than first determining the length of the string. This way, each single character is accessed exactly once (not twice). locale 47.18 (23.12.2020) - Updated the FormatString() documentation to include a warning to the effect that when specifying argument positions (as in "%5$s") only up to 128 arguments may be used. - Rewrote format_uses_position_spec() not to use the index variable i on the string, but to modify the string pointer instead. That made the code noticeably shorter, perhaps faster, too. - Rewrote the initial scanning loop in format_string_with_position_arguments() not to use the index variable i on the format string, but to modify the string pointer instead. Again, that made the code shorter, perhaps faster. - Rewrote format_string_without_position_arguments() to modify the format string pointer instead of using the index variable in on it. locale 47.19 (23.12.2020) - Removed unused code from the format_string_without_position_arguments() function. It will never get called with a "%$s" conversion specification anyway, so we don't have to look for the "$" character. - Reduced the stack size requirements for FormatString() by cutting down the number conversion buffer to 100 characters (from 200). As Patrik Axelsson proved, the worst case never touches the 100 character limit and the recently updated code always checks if it has reached the limits of the number conversion buffer. - Changed format_string_with_position_arguments() so that it dynamically fills the parameter offset array with zeroes as new parameters are tracked. This should be more efficient than always filling the entire buffer with zeroes when the function is invoked. Patrik showed that this approach will significantly reduce the cost of calling this function. - Updated and corrected the documentation (again). locale 47.20 (25.12.2020) - Modified the code which inserts the number grouping characters so that it references the array in locale->el_NumFormat through a local pointer, which generates shorter and maybe faster code. The next step was to rework the code which accesses this array and which ought to make sure that it does not overrun its boundaries. The boundary check is now much simpler in operation and it watches both for over- and underrun. locale 47.21 (2.1.2021) - Permanently enabled the replacement code for FormatString(), which is more performant and has predictable result if internal limits are breached (for example number of reordered arguments). To get an idea of the performance difference compared to the old code, lets show some benchmark numbers for exec.library/RawDoFmt(). The reason for measuring RawDoFmt() is because it is the main use for FormatString(). When locale.library is loaded, it patches exec.library to use FormatString() to handle RawDoFmt(). Some A3000 030@25Mhz benchmarks for RawDoFmt() with plain strings, where: - Iter: Number of iterations of the test - Len: Length of the string used as input to the function - Before: Runtime in seconds for old code. - After: Runtime in seconds for the new code. - Ratio: Ratio between the runtime of the old and the new replacement code. Iter Len Before After Ratio 160 625 1.017 1.049 0.969 800 125 1.186 1.065 1.113 4000 25 2.049 1.213 1.689 20000 5 6.183 1.899 3.255 100000 1 27.027 5.341 5.060 As can be seen, the large initial cost has been reduced significantly. locale 47.22 (9.1.2021) - Mixing conversion specifications with and without a specific argument position with FormatString() was always unreliable and could lead to undefined behaviour by "consuming" more parameter data than was provided. The documentation now reflects this and the implementation tries to bring some consistency to the order in which the parameter data is used. locale 47.23 (15.2.2021) - Turns out that some software, such as TurboCalc 3.5, depends upon the GetCatalogStr() function to return with a specific register configuration. If this configuration does not match how locale.library V38-V40 would set it up, undefined behaviour would follow. locale.library now makes sure that registers D1, A0 and A1 match the locale.library V38-V40 behaviour. Note that D1, A0 and A1 are scratch registers and no code should depend upon their contents unless the documentation states that this configuration is to be expected. - Updated the BUGS section of the GetCatalogStr() function, concerning the accidental A0 and A1 register configuration. - Removed redundant tests from the LoadCatalog() function which saved some 40 bytes. locale 47.24 (16.2.2021) - Small corrections to the documentation in "getcatalogstrwrapper.asm". - Smaller changes to the catalog processing which hopefully result in better code readability. - Updated the catalog processing code to verify if the catalog data could be read and if the catalog contains a valid language chunk. If none of these are present (e.g. the language is missing, is not NUL-terminated or an empty string) then the catalog will be rejected as faulty. Furthermore, if there is no usable version chunk in the catalog and a specific version of that catalog was requested, then the catalog will be rejected as faulty, too (you can't check the version number if the data to check isn't there). Finally, the catalog string information itself is subjected to tests which insure that no zero length strings are permitted, strings can never be larger than what the catalog string buffer holds and numeric overflows before rounding up the total string size are detected early on. The point of this exercise is to curb undefined behaviour. Previously, the catalog processing code assumed that all the chunks were present and trusted that their respective contents were sane. locale 47.25 (7.3.2021) - The debug version of locale.library (which will be the default build until we ship the release version or decide differently) can be instructed to produce debug output via kprintf() if a catalog file cannot be loaded. This is done by setting either a local or global environment variable called "locale.library.debug", like so: setenv locale.library.debug 1 If the variable is not present or contains a value < 1 then no debug output will be produced. Thanks go to Robert Miranda who suggested this feature, which should help in identifying catalog files which do not pass consistency checks or which cannot be loaded for other reasons (e.g. out of memory). locale 47.26 (11.4.2021) - Now handles catalogs with empty STRS chunks robustly, as well as catalog files which lack a STRS chunk. - It is no longer considered an error if a catalog file has no valid FVER chunk or even no FVER chunk in the first place. However, if the client asked for a specific version of the catalog file missing or invalid FVER chunks will be treated as if the catalog version did not match the requested version.