Bug 59551

We need to implement the incremental *updating* of the phone book, too. Otherwise, if the complete download always takes longer than the car is in use, then future updates even to text-only contacts will never get mirrored in the car.

Tom Counihan raised another issue with the current PBAP syncing in private email: if the phonebook download always takes too long, we never process the data and thus the cache forever remains out of sync.

Tom proposed to download in smaller batches and import each batch separately. The problem with that approach is that we still need to get the entire address book from the phone before we can reliably determine deleted contacts.

Let me offer a refinement of the original idea: instead of only
processing a subset of the phone's contacts per sync, let's split the
sync itself into more phases:
      * Start at offset x into the set of contacts.
      * Download n contacts and update the local database with them.
      * Update x persistently.
      * Repeat and role over until we have dealt with all contacts.
      * Then delete all local contacts which had no match on the phone
        anymore.

In other words, we attempt to do a full sync each time we get triggered
to do so, but it won't matter as much anymore when we get interrupted in
the middle, because we will import new or updated contacts eventually.

Deleting contacts will still depend on finishing the full download and
processing.

This batched download can be combined with checksuming by calculating a checksum over each batch. Finding a checksum depends on a few lookup keys:
* start offset
* batch size
* with and without photo data
We need to remove obsolete checksums to avoid growing their number infinitely.

The batched download can be combined with incremental photo download by first downloading batches without photos, then with photos.

(In reply to comment #1)
> We need to implement the incremental *updating* of the phone book, too.
> Otherwise, if the complete download always takes longer than the car is in
> use, then future updates even to text-only contacts will never get mirrored
> in the car.

I implemented incremental photo download for *all* syncs (not just the first one, as originally proposed).

Details are in these commit messages (actual commits not in master yet):

commit 7f47fa336570467a4dd60ada5da41ad302335c36
Author: Patrick Ohly <patrick.ohly@intel.com>
Date:   Fri Jul 5 10:39:21 2013 +0200

    PBAP: incremental sync (FDO #59551)
    
    Depending on the SYNCEVOLUTION_PBAP_SYNC env variable, syncing reads
    all properties as configured ("all"), excludes photos ("text") or
    first text, then all ("incremental").
    
    When excluding photos, only known properties get requested. This
    avoids issues with phones which reject the request when enabling
    properties via the bit flags. This also helps with
    "databaseFormat=^PHOTO".
    
    When excluding photos, the vcard merge script as used by EDS ensures
    that existing photo data is preserved. This only works during a slow
    sync (merge script not called otherwise, okay for PBAP because it
    always syncs in slow sync) and EDS (other backends do not use the
    merge script, okay at the moment because PIM Manager is hard-coded to
    use EDS).
    
    The PBAP backend must be aware of the PBAP sync mode and request a
    second cycle, which again must be a slow sync. This only works because
    the sync engine is aware of the special mode and sets a new session
    variable "keepPhotoData". It would be better to have the PBAP backend
    send CTCap with PHOTO marked as not supported for text-only syncs and
    enabled when sending PHOTO data, but that is considerably harder to
    implement (CTCap cannot be adjusted at runtime).
    
    The original goal of overlapping syncing with download has not been
    achieved yet. It turned out that all item IDs get requested before
    syncing starts, which thus depends on downloading all items in the current
    implementation. Can be fixed by making up IDs based on the number of
    existing items (see GetSize() in PBAP) and then downloading later when
    the data is needed.

commit e9cf7c22fc5c680ee5f1f1d9d0b970933c7098e3
Author: Patrick Ohly <patrick.ohly@intel.com>
Date:   Fri Jul 5 13:42:12 2013 +0200

    PIM: use incremental sync for PBAP by default (FDO #59551)
    
    When doing a PBAP sync, PIM manager asks the D-Bus sync helper to set
    its SYNCEVOLUTION_PBAP_SYNC to "incremental". If the env variable
    is already set, it does not get overwritten, which allows overriding
    this default.

(In reply to comment #0)
> The second part is meant to avoid expensive work (syncing) and thus save CPU
> cycles: by checksumming the downloaded data, it becomes possible to detect
> the case where nothing changed and avoid the syncing.

This has one major downside: the check can only be done after having downloaded all data. In the cases where something changed, the total time then is "download time" + "sync time".

Therefore I intend to not implement checksuming. Instead syncing will run in parallel to downloading and total time will be roughly max("dowload time", "sync time"). If we can make syncing faster than downloading (which is the goal), then we will be done shortly after finishing the download.

(In reply to comment #2)
> Let me offer a refinement of the original idea: instead of only
> processing a subset of the phone's contacts per sync, let's split the
> sync itself into more phases:
>       * Start at offset x into the set of contacts.
>       * Download n contacts and update the local database with them.
>       * Update x persistently.
>       * Repeat and role over until we have dealt with all contacts.
>       * Then delete all local contacts which had no match on the phone
>         anymore.

It could be even simpler:
 * Start at offset x.
 * Download, process in parallel.
 * Download more chunks until done.

However, after discussing the "Offset" value with Luiz van Dentz, our conclusion was that the value is poorly defined: it is not clear whether it is based on the original numbering of contacts (in which case one could ask for contacts #10-#19 and not get any because all of those were deleted) or counts all current non-deleted contacts.

Depending on a feature that is poorly defined, probably not used much and has corner cases which are even rarer and hard to test (contacts deleted while syncing) is asking for trouble. Download times are small enough that the theoretic advantage of downloading in chunks may never matter in practice.

Taking all of that into account, I don't intend to implement downloading in chunks anymore.