OpieSyncProtocol

= Opie Synchronisation Protocol =

Introduction
This document describes the PIM data synchronisation protocol used by Opie in versions 1.2.5 and above, with some reference to differences with older versions.

History
Opie version 1.2.5 has introduced a number of changes in the way PIM data is handled. Most importantly, the SQLite data backends (which were present in previous versions) have been functionally improved and are now used by default. Previous versions stored PIM data primarily in XML, the format of which dates back to Qtopia from which Opie derives. Synchronisation was done by retrieving the entire XML file, updating it and then writing it back, all via FTP. Once XML storage was replaced by SQLite however, a new synchronisation "protocol" needed to be developed.

The "new protocol" is not entirely new - it builds upon the previous synchronisation method, utilising the same transports; the only difference being the addition of some extra QCop calls and the ability to only transmit the changes in both directions rather than the entire database. Allowances are made for synchronising with multiple "peers" (e.g. multiple desktop machines). In addition, backwards compatibility with older clients should still be maintained.

Additionally there is now proper handling for the "memo" (notes) data type. Although this data format is not particularly complex, for synchronisation and storage purposes it is now handled in the same way as any of the other PIM data types.

Components

 * Applications eg. Datebook, Todo List, ...
 * Data access layer (libopiepim2) - library that handles data access. Internally contains:
 * Access classes for each type of data (contact, event, todo, memo)
 * Backends, notably XML and SQLite, for each type of data
 * Changelog - stores information on what records have changed, who has synced and when
 * Launcher (qpe) - main Opie process which is constantly running
 * TransferServer - provides a cut-down FTP server on TCP port 4242
 * QCop bridge - provides access to QCop on TCP port 4243
 * "Server" - class which handles sync-related QCop calls (among other functions)
 * SyncAccessManager - class that wraps the PIM access objects
 * VirtualFS - class which manages "virtual" files provided via the FTP server

Protocol
The transport protocol for data is FTP on port 4242. PIM data is read from "virtual" XML files served by the FTP server whose paths are the same as the XML storage paths in previous versions. The content of these files is read on-demand from the appropriate PIM backend. Non-data commands and responses are sent via the QCop bridge on TCP port 4243.

Synchronisation operates in several different modes - "slow-sync" and "fast-sync" (set separately for both directions). In slow-sync mode, all data is transmitted; in fast-sync mode only the changes since the last sync are transmitted. Slow-sync mode is forced for reading when the changelog has no information about the peer (the first sync), or a peer may request slow-sync mode in the case where the state (or data) on the peer side has been lost. Slow-sync for writing is only useful when the peer explicitly wishes to overwrite all of the data on the device, and it is expected that most clients will write changes back in fast-sync mode even if slow-sync was used for reading. (For compatibility with older sync clients, without making the extra QCop calls, slow-sync is the default for both directions.)

The SQLite backend has a built-in "changelog". An entry is added to the log when a change is made to a record in an application (add, update or delete), as well as when each peer last synced. In this way, once the peer has identified itself the backend can filter the records it provides to include only those that have changed since that peer last synced. The changelog only needs to record only one change per data record since the last sync, and changelog entries that are no longer needed by any known peer are purged automatically; thus storage overhead is kept to a minimum.

Basic operation
The peer (eg. synchronisation software on a desktop machine) authenticates via QCop, connects to the built-in FTP server and then identifies itself and requests to start synchronisation for each application via QCop calls. It then retrieves the data in XML format via FTP, merges the changes into whatever target data it is working with, and then writes its changes (if any) back to Opie by writing to the same XML files over FTP. It then sends a QCop command indicating success, and synchronisation is now complete.

Detailed operation

 * 1) Peer connects to QCop TCP port 4243
 * 2) Peer sends QCop
 * 3) Opie responds with  . The peer should use the version number x.y.z to determine if additional QCop commands (marked NEW below) can be sent (i.e. only if it is 1.2.5 or greater).
 * 4) Peer sends QCop   - this shows the "sync in progress" screen with a specified string on the Opie side.
 * 5) Peer sends QCop   to determine root path (used later when retrieving files via FTP)
 * 6) Opie responds with   where   is a boolean indicating whether the device is locked or not
 * 7) Peer connects to the FTP server on port 4242
 * 8) NEW: Peer sends QCop   where peerid is a unique identifier for the peer (it is suggested that this be randomly generated before the first sync and then stored for use in subsequent sync sessions), and peername is a free-format name of the peer suitable for showing to the user (e.g. the peer's hostname).
 * 9) NEW: Opie responds with
 * 10) For each PIM application type:
 * 11) NEW: Peer sends   where   is the name of the application (,  ,  , or  ) and   is a boolean value which is true to request slow-sync mode for reading and false otherwise;   is the same for writing. (It is expected that   will almost always be false, unless the client explicitly wishes to overwrite all data on the device.)
 * 12) NEW: Opie responds with   where changelog is true if changelog functionality is available (required for fast-sync read); and   is true if fast-sync mode has been enabled for reading or false if slow-sync will be used. (For writeback, the sync mode is whatever was requested.)
 * 13) Peer retrieves XML file for the application using the path   where   was retrieved from   previously. NEW: If fast-sync was available this will contain only the changes since the last sync (each record which is an update will have   and a delete will have only the uid and  . Any record without a   should be assumed to be an add).
 * 14) Peer merges the data with its own as appropriate
 * 15) Peer sends back any changes to Opie by writing them to the same XML file over FTP - if slow-sync writeback was requested then all data should be written. NEW: if fast-sync write was requested then only the changes should be written (with the same   labelling as with fast-sync for reading described above).
 * 16) NEW: If all has succeeded, peer sends QCop  . This tells Opie that everything has succeeded, and it will then save the "last sync" information for the peer and purge any changelog entries that are no longer needed.
 * 17) NEW: Opie responds with
 * 18) Peer sends QCop   and disconnects.