notes.txt

Q - how to tell difference between sensor and subject in rep file?
    A - all uncommented lines will refer to platforms
    A - sensors defined by ;SENSOR pragmas?
Q - does everything get added to state table?
    A - all vessel tracks yet, contacts and activations can come from pragmas
Q - how to add entries to Entry table that appears as foreign keys in several other tables
    A - although all uuid keys have a default function to generate a uuid, need to create one in entry table first and use that
Q - All the table names in postgres seem to be quoted, could they just be plain names?
Q - Is it ok to use constants for the TableTypes? (I've just modelled them as data members of each DB class)
    Or does it need a mechanism to automatically lookup or insert table type?
Q - Not all tables with UUIDs seem to be foreign keyed to Entry table? eg. DatafileTypes, Nationalities both have a UUID but it's not forced to belong in the Entry table
Q - Should I validate string lengths for DB fields against file entries? eg. DB has length of 150 for some fields
Q - when looking for matching Datafile, should I use 'reference' for filename? or always just add every time? should this be just the filename or a full path?
Q - Main confusion: State table just refers to sensors, not to platforms. Although lines in REP files are all platform movements, not sensor movements
    This seems to indicate that the State tables is a list of Sensor values, but I'm thinking they are position changes of a vessel/platform. Where do the vessel/platform positions get tracked?
    Or are they just tracked in relation to the Sensor that's keeping track of it?

- ideas:
  - when seeing foreign refs in file need to check if already known in memory. DONE - Datstore caches
  - if not known in memory search for it in DB. DONE - query methods when adding elements
  - if not in DB see if can be created for known or contextural info to create new entry. DONE - create new rows
  - if not look for data in .ini files to create new entry
  - Use with: or some other mechanism to start transaction, and always commit or rollback at end without explicit calls. DONE - use context manager session_scope

- poc limitations:
  - just long/lat to indicate position
  - hardcoded refs to other tables (for now) DONE now uses data resolvers

- sqlite + missing data resolvers
  - sqlite integration seems ok, but would be worried that types dont line up well or fuller integration not tested fully
  - need to work out how to make postgres and sqlite integrations co-exist with as much common code as poss, diff DB model classes?
  - add DefaultsResolver to use when no other resolver specified, which would set opinionated defaults, good for empty DB and means can do away with hardcoded default DB entries
  - have added more questions to CommandLineResolver to let it populate a completely empty DB
  - add caches for all types

  + TODO before marking done
  - method for common function pattern
  - fork of DB classes into Sqlite and Postgres vers DONE
  - try on a big file DONE
  - fix parse error for platform name in quotes
  - add relationships/foreign key defs
  - try and UUID stuff common inside of Datastore DONE

- used packages with Python 3.6.4
  - SQL Alchemy 1.3
  - geoalchemy2
  - psycopg2
  - nose2 (0.9.1)

- run from command line with module flag and no file extension eg:
  py -m Experiments.DataStore_resolverExperiment
- make sure all module imports done relative to top level qualified with package name eg: Store.DBBase