TABLE OF CONTENTS xpksub.library/XpksPackerInfo xpksub.library/XpksPackChunk xpksub.library/XpksPackFree xpksub.library/XpksPackReset xpksub.library/XpksUnpackChunk xpksub.library/XpksUnpackFree xpksub.library/--sublibs-- xpkmaster.library/--general-- xpkmaster.library/--general-- GENERAL COMMENT: There is a lot unimplemented stuff. When you do not find a proper documentation, do not use it. XPK is a standard packer system, where xpkmaster.library does the whole managment stuff and xpkXXXX.library's do the packing and unpacking stuff. The following should explain how such a library is build. XPK does its packing with chunks. This means not the whole data is crunched in one pass, but part for part. This allows to do progress reports and even to decrunch large files when you have less memory. So you always need to set the flags XPKIF_PK_CHUNK and XPKIF_UP_CHUNK. There are ARCHIVE and STREAM flags as well. This is future stuff and not used at the moment. Each library has 101 (0-100) different modes, which can be selected by user. You either can produce one XpkMode structure using these 101 modes (or even ignore them) or you may divide it into different XpkMode structures (you need to set XPKIF_MODES flag). Anyway the line of XpkMode structures must start with mode 0 and end with mode 100. If you support different modes, a higher mode means always better crunching factor. The XpksPackerInfo functions tells xpkmaster.library all necessary information about a sub library. It is called after the library is loaded. The xpkmaster.library stores the crunched length, uncrunched length and a checksum of crunched data for every chunk, so the sub library does not need to do that. The sub library function XpksPackChunk is called for the first chunk till the last chunk. These chunks may have different size, but it is guaranteed that the first one is always the biggest. If an error occurs or the file is finished, XpksPackFree is called. Same happens for decrunching with XpksUnpackChunk and XpskUnpackFree. The crunched data may depend on previous crunched chunks (e.g. a table stored in chunk 1). So if we start a new crunch, XpksPackReset is called to end this current one. Using XpkPackReset xpkmaster can force the sub library to produce independent chunks. NOTE: It is allowed to do such stuff depending on previous chunks, but as seeking is disabled by that, it's not recommended. You need to set XPKIF_NOSEEK flag for such clients. Libraries supporting seek normally have an empty XpksPackReset function. When your library supports encryption, you need to set XPKIF_ENCRYPTION flag, when your library always needs a password, set XPKIF_NEEDPASSWD as well. When the library does not preserve original file (it is lossy like JPEG), then set XPKIF_LOSSY flag. To make XPK really save, you get a safety margin. This means for crunching the destination buffer is at least XPK_MARGIN bytes longer than the input file - For decrunching the destination buffer is at least XPK_MARGIN bytes longer then the decrunched data. All information is transfered through XpkSubParams structure. Most important are xsp_InBuf, xsp_InLen, xsp_OutBuf, xsp_OutBufLen. xsp_InLen is size of input file for packing and size of crunched file for unpacking. xsp_OutBufLen is always the real buffer length. This includes the XPK_MARGIN value. Never exceed the buffer borders. XPKERR_EXPANSION error code exists for such cases. The chunksize information in you library information helps xpkmaster.library to find the used chunk size (It's not always the user selected value). Minimum and maximum values depend on your algorithm. The Default value should be something around 50KB or a value which brings fine results with your packer. See below for additional comments. xpksub.library/XpksPackerInfo xpksub.library/XpksPackerInfo NAME XpksPackerInfo - Get information about a packer sub-library SYNOPSIS info = XpkPackerInfo() D0 struct XpkInfo *XpkPackerInfo(void); FUNCTION Returns all information about a packer sub-library for use by the master library. RESULT info - pointer to struct XpkPackerInfo. see xpk/xpksub.h for the meaning of all the fields. SEE ALSO XpksPackChunk() xpksub.library/XpksPackChunk xpksub.library/XpksPackChunk NAME XpksPackChunk - Pack a chunk of data SYNOPSIS err = XpksPackChunk( param ) D0 A0 LONG XpksPackChunk(struct XpkSubParams *); FUNCTION Packs one chunk of data from memory to memory. The size of the chunk will not surpass the MaxPkInChunk field of the packer info structure. Chunks are numbered in case a library writes various dictionaries at special places. INPUTS struct XpkSubParams, where the fields mean: InBuf Pointer to the data to be compressed InLen The number of bytes to pack OutBuf Pointer to the memory area to write packed data to OutBufLen Size of above memory area OutLen Will be set to compressed length by sublib Number The number of this chunk Mode The sub-mode to be used Password The password to be used for encryption Sub[4] Here the sub library stores its private data RESULTS err - global xpk error code OutLen - Number of bytes written SEE ALSO XpksPackReset(), XpksPackFree() xpksub.library/XpksPackFree xpksub.library/XpksPackFree NAME XpksPackFree - Free buffers associated with packing process SYNOPSIS XpksPackFree( param ) A0 void XpksPackFree(struct XpkSubParams *) FUNCTION Frees all buffers the sub-library has allocated privately while doing some packing INPUTS param - The XpkSubParams used for packing SEE ALSO XpksPackChunk(), XpksPackReset() xpksub.library/XpksPackReset xpksub.library/XpksPackReset NAME XpksPackReset - Clears all state information SYNOPSIS XpksPackReset( param ) A0 LONG XpksPackReset(struct XpkSubParams *) FUNCTION Clears all internal tables and any other state information of the packer. This has the effect, to guarantee independent unpacking of the chunk to come. The packers call this function automatically before returning XPKERR_EXPANSION (data could not be compressed). Note: packers have to store all of their state relevant to packing in XpkSubParams to ensure reentrantness. INPUTS param - The XpkSubParams used for packing SEE ALSO XpksPackChunk(), XpksPackFree() xpksub.library/XpksUnpackChunk xpksub.library/XpksUnpackChunk NAME XpksUnpackChunk - Uncompress a chunk of data SYNOPSIS err = XpksUnpackChunk( param ) A0 LONG XpksUnpackChunk(struct XpkSubParams *) FUNCTION Decompresses one chunk of data. INPUTS param - The XpkSubParams used for unpacking xsp_InBuf Pointer to the data to be uncompressed xsp_InLen The number of bytes to unpack xsp_OutBuf Pointer to the memory area to write unpacked data to xsp_OutBufLen Size of above memory area xsp_OutLen Must contain the decompressed size of the data xsp_Number The number of this chunk xsp_Password The password to be used for decryption xsp_Sub[4] Here the sub library stores its private data RESULT err - global xpk error code SEE ALSO XpksUnpackFree() xpksub.library/XpksUnpackFree xpksub.library/XpksUnpackFree NAME XpksUnpackFree - Free all private data the sublib has allocated SYNOPSIS XpksUnpackFree( param ) A0 LONG XpksUnpackFree( struct XpkSubParams * ) FUNCTION Will free all memory the sub-library has allocated during the decompression. INPUTS param - The XpkSubParams used for unpacking SEE ALSO XpksUnpackChunk() xpksub.library/--sublibs-- xpksub.library/--sublibs-- Some remarks for sublib writers ------------------------------- First of all, Read the documentation. If you have got any further questions writing xpksublibraries, either contact one of the XPK maintainers. The name of a sub library is xpkXXXX.library, where XXXX is a combination of uppercase characters ('A' to 'Z') and numbers ('0' - '9'). The first character must not be a 'X' (xpkXTRA.library is not allowed), as this letter is reserved for xex libraries. How XpkPackChunk works: - Check xpar->xsp_Sub[0]. If it is 0, this is the first call. Allocate your tables and store pointers to them in xsp_Sub[0] .. xsp_Sub[3]. - Read xpar->xsp_InLen bytes from xpar->xsp_InBuf and pack them to xpar->xsp_OutBuf. Do not exceed xpar->xsp_OutBufLen. Store packed len in xpar->xsp_OutLen. How XpkUnpackChunk works: - Check xpar->xsp_Sub[0]. If it is 0, this is the first call. Allocate your tables and store pointers to them in xsp_Sub[0] .. xsp_Sub[3]. - Read xpar->xsp_InLen bytes from xpar->xsp_InBuf and unpack them to xpar->xsp_OutBuf. Do noot exceed xpar->xsp_OutBufLen. Maybe compare uncompressed len with xpar->xsp_OutLen. Other notes: - Your library should, of course, be re-entrant. This means no PC-relative addressing of writeable data. Whatever you have, store a pointer to it somewhere in the XpkSubMessage. In short: All state information of the sublib is in the XpkSubMessage. You can detect the first chunk of a file by the fact that this pointer is still NULL. In case your lib is not re-entrant, return XPKERR_LIBINUSE when required. - There will be no larger chunks on (de)compression than the first one. (But with Seek you may not get first chunk really as first!) - On compression, you may expand the data by XPK_MARGIN bytes maximum. This means the output buffer is as large as the input buffer plus XPK_MARGIN. Compression libs would, of course, already return XPKERR_EXPANSION if they exceed the input buffer. - You must supply a function XpkPackReset, which clears all tables so that the next chunk will be able to be unpacked independently. You also have to reinitialize all state information [calling your XpkPackReset] before returning XPKERR_EXPANSION! - On decompression, you will also have a XPK_MARGIN byte saftey margin for runaway unpacking. - Fill in one XpkInfo structure. See xpk/xpksub.h for the meanings of the fields therein. - If your packer has only one packing mode, fill in one XpkMode structure (see xpk.h). Set the 'xm_Upto' field to 100, this means this struct handles all modes up to (and including) 100, which is the max packing mode. Place a pointer to this struct in the XpkInfo structure. - There must be at least one XpkMode structure, and the last XpkMode structure must have the xm_Upto set to 100. It must! - If your packer has several modes, create several XpkMode structures and sort them by efficiency in ascending order. Place a pointer to them in XpkInfo. Set the XPKIF_MODES flag in the xi_Flags field of XpkInfo. - If your packer cannot decrunch the chunks in any order, set XPKIF_NOSEEK flag. - Your packer will get the 0..100 number and is responsible of mapping it to its own packing modes. - Set the xi_DefMode field to some 0...100 number. - If XSF_STEPDOWN is set in the xsp_Flags field of the SubParams structure, you are allowed to reduce packing efficiency in order to save mem. - Encryption libraries should compute a cryptographic hash [that is a checksum] while or after decryption and compare it with one stored in the encrypted data. If the hash of the decrypted text does not match the stored checksum, return XPKERR_WRONGPW. Note, that this hash inevitably discloses information applicable to breaking the code. Try to keep this information small, e.g. by using a 16bit hash. Take care, that one has to decrypt the whole chunk before it is possible to compute the hash of the decrypted data. How to get values for xm_PackSpeed, xm_UnpackSpeed, xm_Ratio? ------------------------------------------------------------- Older versions of xpkmaster.library defined an Amiga A3000/25 with SCRAM and AmigaVision executable as standard test for pack algorithms. I searched for better testing defines and defined a new system which should be used now and in future: At http://corpus.canterbury.ac.nz/ you find a big set of files called "Canterbury Corpus". (Not the large one, but standard test archive). This is a collection of standard test files which can be used to test the packing algorithms. Speed values are still a problem, as it is hard to get a value usable on all machines. New values should be computed on an A4000 with 68060/50Mhz with no startup-sequence bootup. This defines a big field of different machines which differ in RAM size, bus speed and other parameters, but the test must be usable and so I cannot define one special environment. The values entered in XpkMode structure are a sum of all test files. You need to pack all files and note their results (reached file size, crunch time, decrunch time). Use xBench to do that, as this is very accurate (as good as possible). Now add input file size (INSIZE), crunched file size (CRSIZE), needed crunch time (CRTIME) and needed uncrunch time (UNCRTIME) for all of the standard test files! xm_Ratio = 1000*(1-(CRSIZE)/(INSIZE)) xm_PackSpeed = INSIZE/1024/CRTIME xm_UnpackSpeed = INSIZE/1024/UNCRTIME When you do not own an A4000 with 68060 CPU at 50Mhz then try to adapt speed values.