Roger Mathew's TAS 3 Jottings

Main TAS page           Dates TAS Data Dictionary ReStructure TAS data to ASCII

Here are some of the things I've discovered about TAS over the years.

Number fields

Before TAS-Pro 4 (which uses IEEE numeric format), TAS numeric fields hold Binary Coded Decimal (BCD) integers. The position of the decimal point is determined by reference to the data dictionary field DICT.DEC. Each byte of the numeric field represents two decimal digits. For odd-sized fields, the first half-byte holds either F (hex), indicating a positive or Eh, indicating a negative value. The decimal value of each subsequent digit is the hexadecimal value of each subsequent half-byte. Thus: For even-sized fields, the whole field is used, which is why you have to create odd-sized fields if you want to store negative numbers.
Top

Dates
(North American readers, please note that I am using the British date convention: dd/mm/yyyy.)

TAS holds dates internally as 8-digit integers. 1 Jan 1860 is held as 2400411. Every 256 years, dates between 1 Jan and 29 Feb display "off" by -256 years. The date next after "31/12/2047" displays as "1/1/1792" and this anomaly continues until "29/3/2048" (displayed as "29/3/1792"), whereafter the date next following displays as "1/3/2048".

This also happens in 1792, where 1/1/1792 to 29/2/1792 display as 1/1/1536 to 29/3/1536, in 1536 (first 2 months as 1280) and 2304 (first 2 months as 2048).

I guess this isn't going to worry any of us much, especially as it seems to have been corrected in TAS 5.1. It probably arises from an erroneous byte-shift in the leap year routine.

Top

Data Dictionary

TAS data files contain no field information. BTRIEVE files don't. All the data about file structure are held in the data dictionary, TASDICT.M. The only thing that BTRIEVE knows about are the record length and the offsets, lengths and types of index (key) data.

TASDICT.M holds information about field names, sizes and types, which fields are keys and what the key types are. TASDICT.M also holds data about impending changes to the file structure, to enable files to be restructured. TASFILE.M holds data about data files. You can have several files with identical structures, referencing the same layout ("schema") in TASDICT.M. TASFILE.M links any particular file by its filename to the its schema in TASDICT.M.

The fields in TASDICT.M are as follows:

DICT.NAMEField name
DICT.SCHEMA  Schema nameusually the file name
DICT.SKEYan overlaid key
DICT.OFFSETOffset of the start of this field from the start of the record  1-based
DICT.TYPEAlpha, numeric, date, etc
DICT.SIZEDisplay size
DICT.DECNumber of decimal places
DICT.UPCASEForce uppercase if "Y"
DICT.HOLDERCopy of DICT.ARRAYused by ReStructure routine
DICT.KEYKey type
DICT.KEYNKey number
DICT.OVLYIndicates an overlay field if "Y"
DICT.SIZEHSize in bytes (not necessarily the same as display size)
DICT.ARRAYNumber of array elements (if any)
DICT.DECHCopy of DICT.DECused by ReStructure routine
DICT.SIZEHLDCopy of DICT.SIZEHused by ReStructure routine
DICT.OFFHCopy of DICT.OFFSETused by ReStructure routine

When a TAS program is compiled, the compiler reads TASDICT.M and converts field names into offsets. TASDICT.M is also referenced by many of the TAS utility programs to get field data on the fly when maintaining or browsing data. It is not used at run time by compiled programs unless the program calls for it (eg many of the TAS system programs).

Top

ReStructuring Data Files

Array fields in a file structure give the internal TAS ReStructure routine serious indigestion. This has been fixed in version 4.xx onwards. The effect of it is that you will lose data in array fields if you ReStructure a file.
Here is how RS is supposed to work:
  1. When you create a file structure, TASDMGR makes entries in TASDICT for each of the fields as shown in the section on TASDICT.M.
  2. When you create the file, TASDMGR calls a BTRIEVE routine and feeds it the data that BTRIEVE needs to create the file. It then updates the TASDICT.M records by copying offset and size data for each field into "holder" fields (DICT.HOLDER, DICT.DECH, DICT.SIZEHLD, DICT.OFFH), thus recording the last-initialised state of the file structure.
  3. When you change the structure of a file, TASDMGR does not update the "holder" fields. Thus, TASDICT.M will hold both the current state of the file and the new structure that will be implemented by the ReStructure routine.
  4. The ReStructure routine uses data from the "holder" fields to find the correct offsets from which to read data from the existing file. It then calls the BTRIEVE "create" routine, which creates a new file, having the new structure.
  5. Having copied data into the newly created file, RS then updates the "holder" fields in TASDICT.M, so that it can effect a further restructuring later if required.
Unfortunately, the RS routine in TAS 3.xx fails to update DICT.HOLDER, which is supposed to hold a copy of DICT.ARRAY. Instead, it merely zeros it.
Any subsequent ReStructuring will corrupt data in many cases. Specifically, if field sizes are enlarged or the number of array elements increased (or both), data from the previous file structure are left behind, thus corrupting the contents of the enlarged field or new array element. If numeric data are left at an offset now reserved for alpha or vice versa, you get some very odd-looking results.

I wrote a fix for this, which I have used to avoid further problems. If you want to know how it was done, read on, else you can skip over the following account, taken from the documentation that I send out with the FIXRS package.

        To  fix  these problems,  it was necessary first to write  FIXRS, 
        which modifies the DICT.HOLDER field to agree with the DICT.ARRAY 
        field.   However,  it  was  also necessary to  ensure  that  file 
        definitions  in  which any existing value of DICT.HOLDER did  not 
        agree  with  DICT.ARRAY  were  left  unmodified,  as  this  would 
        indicate that a ReStructure was pending (TAS allows and ought  to 
        allow  successive  modifications of Dictionary  between  physical 
        restructuring  of  files).   This  was  accomplished  by  setting 
        DICT.HOLDER  to 1 if it was currently zero.   A preliminary check 
        for  a DICT.HOLDER value greater than zero is thus  effective  in 
        ensuring  that a file is not subjected to FIXRS when it ought not 
        to be.

        Provided no existing DICT.HOLDER values are greater than zero, it 
        is assumed that the file has not been FIXRSed,  and each relevant 
        DICT.HOLDER value is set to equal DICT.ARRAY.  This sets the file 
        in  a  suitable  state  to  perform  correctly  during  the   TAS 
        ReStructure (RS) routine.

        It  was also necessary to ensure that a similar routine was built 
        in   to  the  Create  Database  (TASADFLE)  program  after   file 
        initialisation  because  initialisation wrongly  resets  all  the 
        DICT.HOLDER  fields to zero.   The same routine was conditionally 
        built into the Maintain Data Dictionary (TASDMGR) program for the 
        case when the file is created by TASDMGR,  and only in that case; 
        subsequent  changes to an existing structure must of  course  not 
        result in updating of DICT.HOLDER.

        Finally,  MENU.RUN  had  to  be modified to run  FIXRS  following 
        either  a File Initialisation (FI) or a ReStructure  (RS).   This 
        was more difficult,  as TAS programs do not return to the calling 
        program.   I  have still not succeeded in having FIXRS know  what 
        file has been restructured or initialised,  so I cannot make  the 
        process entirely transparent.

        Before  ending  this  discussion of  ReStructure,  the  following 
        points should be noted:

        1.   RESTRUCTURE will not handle an attempt to change the TYPE of 
             a data field.   It was never intended to.  If you attempt to 
             convert  a  numeric field to alpha,  the effect will  be  to 
             increase the record size,  so there will be no  difficulties 
             (mutatis mutandis) with spurious record generation,  but the 
             data  will  effectively have been lost.   The  actual  alpha 
             field  will  hold whatever bytes were in the numeric  field, 
             and be right-padded with spaces (ASCII 32d,  20h).   If  you 
             attempt to convert an alpha field to numeric,  you will most 
             likely  end up with spurious pseudo-numeric "data" (often  a 
             lot of "20"s) in the resulting field.   The way to deal with 
             this  (if  you  need to),  is to define a new field  of  the 
             appropriate  type  (ie numeric if you want to  go  alpha  to 
             numeric),  restructure,  write  a short program containing a 
             fragment like:

                          find(B,keyfld,flerr)     ;get first record
                     loop:                         ;while not end of file
                          alfld = numfld           ;do the business
                          save(flnam,n,n)          ;save it in the file
                          find(N,flnam,eof)        ;get the next record
                          goto loop                ;and repeat
                     eof:

             Then, after checking your results, redefine the structure to 
             remove the unwanted field, restructure again and it's done!

        2.   When a new Key field is added to a file structure,  it  does 
             not   become  effective  until  AFTER  the  file  has   been 
             ReStructured.   In  this respect,  TAS Professional  differs 
             from TAS-Plus, which required explicit re-initialisation and 
             reindexing  after  ReStructure in order to implement  a  new 
             key.   YOU  MUST  NOT re-initialise a TAS Professional  file 
             unless you are prepared to abandon all its data!  If you re-
             initialise,  you lose the data permanently,  in contrast  to 
             the  TAS-Plus arrangement,  which left the data file intact.  
             The  TAS  Professional ReStructure routine involves  a  file 
             initialisation, which is done prior to reading back the data 
             which have been written out to a holding file (which is then 
             deleted).
Top

Writing out TAS data to ASCII files

I am often asked for an easy way to write out the data in TAS files to an ASCII text file. It isn't all that easy if you have only the TAS system programs and all but impossible if you don't. You definitely need the Data Dictionary files (TASDICT.M and TASFILE.M). Here's the theory:
Top

Number fields Dates TAS Data Dictionary ReStructure TAS data to ASCII           TAS main page
7 March 2004