Skip to content

X Sync Model

Jump to bottom
davidbannon edited this page Dec 30, 2018 · 4 revisions

Still trying to reverse engineer the Tomboy Sync Model

Note : October, 2018. Sadly, even this model does not hold up as I require it to. Indeed, it does explain some of Tomboy's problems during sync but thats not its purpose. So, until I get around to rewriting it, please disregard it ! If you must know how tomboy-ng does its sync, please look in the source directory, sync.pas Most of the tricky stuff is implemented there, there is no GUI, and there is a reasonably complete text description at the top of the file.

Davo

Found a presentation, slide 22 that appeared to indicate that the Tomboy Sync worked in a different order than I thought. And different from how tomboy-ng's file sync is implemented. This might be a much better way of doing Network Sync and help us understand how the Servers like Rainy might behave. So, filling in the gaps, this looks pretty good. The slide says, and I quote, the sync is like subversion and -

  • There is a running version number for the notes on the server
  • Local copy of notes are like working directory
  • When sync starts Tomboy requests changes since the latest version in its local repo (sadly, further analysis says that this statement is misleading. Perhaps the slide author was simplifying ..)
  • Download changes - apply against local copy (svn up)
  • Upload local changes (svn commit).

So, I postulate that the client asks the server for a list of notes, new since the client's last sync (which the client nominates). The client knows the date of its last sync and its last revision number. The list comes back with (at least) the note ID and the revision number applying to that note. This process then happens -

Flow Chart

(See note below about the "rejoining a sync" problem.) Once completed above, the client needs to

  • Delete notes listed in current local manifest but not in remote one. These have been deleted by other clients.
  • Upload the notes that are not mentioned in its existing local manifest. They have not been seen by the sync system before. Make reference to them in the new local manifest along with the new Rev Number.
  • Then update its local manifest. The local manifest should, at the end of the sync process, list all local notes present in the notes directory along with their actual revision numbers (why ?). It should list no deleted notes.

Issue - get a list, from server, of all notes it has, a copy of remote manifest in other words, and make our own decisions about which ones to check ? The remote server need only fill in last-change-date for notes after nominated revision number so not a massive time penalty. But we need all remote note IDs so we can check what notes have been deleted by other clients.

The server.

The server needs to (apart from initialization and authorization stuff) -

  • Provide a list of all notes in repo, ID and Revision Number and, in the case of notes with a later revision number than the Client's, its last-changed-date.
  • Deliver a requested note.
  • Accept a list of uploaded notes.This will increment revision number.
  • Accept advise that a note (or list of notes) has been deleted locally. This will increment revision number.
  • Maintain, in what ever way it chooses its current rev number.

I do not know if it makes sense to combine all activities that inc rev no or better to let lots of number happen.

The Client

Writes a new local manifest unless nothing happens. By nothing I mean none of -

  • New file down loaded (add that file to list).
  • Advised server of a delete (can then empty delete section).
  • New file uploaded (add that file to main list). A synced client also needs to maintain the local manifest during day to day operations, if a (synced) note is deleted, it must be added to the Deleted section.

The Manifests

  • Firstly, the two manifests are quite different, even using different syntax for similar data.
  • Both store a Server ID, and a revision number. The Local Manifest also has a date of last revision. Obviously the Client Local Manifest can have a revision number equal to or less than the Server. The ID is to ensure you are talking to the right server.
  • The Local Manifest, stored next to the config file, records the ID of locally existing previously synced notes and (in a separate block) the ID of previously synced notes that have now been locally deleted. Its updated at each sync and whenever a previously synced note is deleted in the client. The list of deleted notes is flushed after each sync. The synced note entries also includes the latest revision number (but I am unsure why).
  • The Server or Remote manifest lists all the notes it has in it's 'database'. Along with the most recent revision number. Here the revision number is important, in file sync at least (where revision number corresponds to a directory), it tells us where to look for a file.

The "rejoining a Sync" Problem

I (David) believe there is a problem in the Tomboy model that shows up when you re-join a sync. By definition, if you are joining a sync, your revision number is zero. But even if you do have notes in your local directory with same ID as ones in the remote repo, the server will indicate you should download every note it it has, based on revision numbers. This is how you end up with a lot of duplicate notes.

The Tomboy File sync does not store last-change-dates in the remote manifest. At present, we are considering adding a last-change-date into the remote manifest in the hope that Tomboy will ignore it and tomboy-ng will be able to do smarter and more flexible jumps between different repositories.

In the case of the one network sync thats under development, its not intended to be Tomboy compatible so not an issue.

Implementation

At this stage, we identify 5 object layers.

  1. (most of) Application, eg tomboy-ng
  2. The upper part of my sync, the bits that work with 1.), its syncgui in tomboy-ng.
  3. Part of my filesync, the bits that claim to know how to sync. Much of the logic, no file handling.
  4. Part of my filesync, the bit that touch the network. Does local file handling.
  5. eg the part of my filesync but this time that writes to the remote repo, in a network sync, this is the remote server.

So, 4.) and 5.) communicate over the network. When the user switches between file sync and network sync, they from layer 4.) and 5.).

Layer 3, in tomboy-ng does not change when changing between netsync an file sync, Layer 4 does is quite different. In a Network Sync Layer 4 communicates over the network to layer 5, the remote server. In a filesync model, we'll keep the layers clear but obviously all are part of the same binary and source tree. But different pascal Units.

Layer 4 functions made available to (eg) Layer 3.

  • function TLevel4.GetNewNotes(NoteMeta : TNoteMetaList; LocRev : integer) : boolean; // Request a list of new notes since nominated revision with, ideally, last-change-date for each. Parameter is a List of Records,[ID, RevNo, LastChangeDate]
  • function TLevel4.DownloadNotes(const DownLoads : TStringList) : boolean; // Request that all the notes mentioned in the simple list be downloaded and, if necessary, an existing note be moved to Backup.
  • function TLevel4.DeleteNote(const ID : string) : boolean; // Advise server that a note has been deleted (does this trigger a rev ??). Can it be a list of notes ? Does this trigger a new revision.
  • function TLevel4.UploadNotes(const Uploads : TStringList) : boolean; // Push a note, or ideally, a set of notes up to server (triggering a new release).

Layer 4 interaction with Layer 5 Layer 4 will initiate communications, over the network, with the remote server in the case of network Sync. With file sync, it may well write directly to disk ....

Warning - totally ignoring initialization, authorization, locking and race conditions.

Spec for implementation

Work on an experimental sync is underway.

https://github.com/tomboy-notes/tomboy-ng/tree/master/experimental

Below are notes on its scope and responsibilities.

Ruby script get_new_notes function

Purpose

Returns a list of notes which have a remote manifest rev larger than the local server's local manifest's rev

Input

In the Pascal stub that calls ruby, there will be a line like -

RunCommand('ruby sync.rb get_new_notes 212')

Output

A stream of data to $stdout: NoteID, RevNo, LastChangeDate for potentially many notes:

0B430A52-C425-4E7F-8A0C-E259468BD0AB, 213, "2018-08-16 12:43:333+10" .....

Ruby script CRUD functions

Based on the output of get_new_notes, the Ruby script will be called again for each new required action. Actions can be:

  • Upload (Nextcloud#upload - uses versioning, is both Create and Update in one function)
  • Download (Nextcloud#download)
  • Delete (Nextcloud#destroy)

Documentation for the Nextcloud gem that facilitates CRUD can be found here: https://github.com/dachinat/nextcloud

Input

RunCommand('ruby sync.rb upload 0B430A52-C425-4E7F-8A0C-E259468BD0AB')

RunCommand('ruby sync.rb download 0B430A52-C425-4E7F-8A0C-E259468BD0AB')

RunCommand('ruby sync.rb delete 0B430A52-C425-4E7F-8A0C-E259468BD0AB')

Output

True or error code, depending on whether the server action happened or returned an error. This still needs working out.

Clone this wiki locally