---- Reported by jingke.zhang@intel.com 2010-04-13 01:57:49 +0000 ----

This is from http://bugzilla.moblin.org/show_bug.cgi?id=4611

   Description  From  pohly   2009-07-21 07:59:27 PST   (-) [reply]

It is a very common complaint that SyncML in most installations only supports
one calendar, address book, task list, memo list per device. This is not a
limitation of the protocol, more like the lowest common denominator that is
typically supported.

Our own sync UI for Moblin follows the same model, although SyncEvolution
itself could handle more than one instance of each data category if the peer
did (evolutionSource config option).

For those cases where the peer doesn't, it would be nice if SyncEvolution could
join data from multiple databases if the peer supports only one database, then
when receiving updates back correctly sorted them into the the right database.
Here's a user giving more background information:
http://article.gmane.org/gmane.comp.mobile.funambol.user/11522

ScheduleWorld has such a feature for calendars, based on the CATEGORY property.
We could adapt a similar system and perhaps even make it so that n:n calendar
synchronization within the same sync session becomes possible.

Another use case for such a join/dejoin are devices which exchange events and
tasks via one single URI. For our direct-sync work we need to support that.

The Synthesis engine has the concept of a <superstore>: it automatically merges
and splits based on some configurable property. Currently this is used to put
VEVENT into one <datastore>, VTODO into another. Perhaps this can be configured
to use CATEGORY, too.

For joining items, make sure that a unique CATEGORY is set per database. For
receiving items or updates back, check for the presence of those categories and
store accordingly, with one database as default for new items which don't have
any of these categories.

------- Comment #1 From pohly 2009-10-29 00:47:48 PST (-) [reply] -------

Besides joining/dejoining different data sources of the same content type based
on some item property like CATEGORY, a much more urgent (and simpler) use case
is accessing events and tasks under one URI.

Apparently ovi.com (https://bugs.meego.com/show_bug.cgi?id=3182) expects its clients to support this and thus
only offers one URI for both kinds of items. It's debatable whether the server
shouldn't better support separate URIs (because there are clients which only
support that).

Our own server mode definitely has to support a joined event/task URI.

I'm not creating two issues for this because the implementation of the two use
cases clearly is related. For tracking purposes: joined events/tasks are P2 for
1.0, the rest can wait.

------- Comment #2 From Chen Congwu 2009-11-19 21:01:21 PST (-) [reply] -------

(In reply to comment #1)
And to sync with Nokia phones, we also need this feature.
FileSyncSource can handle both sync types but EvolutionSyncSource not.
We need to configure and join Evolution events and todos. 
My suggestion on the configuration file change:
Add a "super calendar" = "super evolution calendar" special backend, which will
map to (calendar+todo), the format used is vcard2.1 and ical2.0 by default, but
user can change it:
super calendar:vcard3.0+ical2.0

> Besides joining/dejoining different data sources of the same content type based
> on some item property like CATEGORY, a much more urgent (and simpler) use case
> is accessing events and tasks under one URI.
> 
> Apparently ovi.com (https://bugs.meego.com/show_bug.cgi?id=3182) expects its clients to support this and thus
> only offers one URI for both kinds of items. It's debatable whether the server
> shouldn't better support separate URIs (because there are clients which only
> support that).
> 
> Our own server mode definitely has to support a joined event/task URI.
> 
> I'm not creating two issues for this because the implementation of the two use
> cases clearly is related. For tracking purposes: joined events/tasks are P2 for
> 1.0, the rest can wait.

------- Comment #3 From pohly 2009-11-20 00:15:22 PST (-) [reply] -------

(In reply to comment #2)
> And to sync with Nokia phones, we also need this feature.
> FileSyncSource can handle both sync types but EvolutionSyncSource not.
> We need to configure and join Evolution events and todos. 
> My suggestion on the configuration file change:
> Add a "super calendar" = "super evolution calendar" special backend,

The solution of this problem should be independent of the Evolution backends.
The general task is: "combine two (or more) real data sources into one virtual
one".

I'm not sure whether we really need to write a backend. At least for the "tasks
+ events" use case, the Synthesis engine already has a solution. We just need
to activate it somehow.

> which will
> map to (calendar+todo), the format used is vcard2.1 and ical2.0 by default,

Why vcard2.1?

------- Comment #4 From Chen Congwu 2009-11-20 00:30:47 PST (-) [reply] -------

(In reply to comment #3)
> (In reply to comment #2)
> The solution of this problem should be independent of the Evolution backends.
> The general task is: "combine two (or more) real data sources into one virtual
> one".
Now comes this solution:
The configuration of type properties is defined as:
backend1+backend2+...backendN:format1+format2_...formatM (0=<M<=N)
The information we lost is: the datastore name is the same as backend name in
such joined case which I think is OK.
Another dirty point is the "forced" property in format, the original single
m_forced property is no longer working.

We will maintain a list of all joined datastores, which typically like:
superdatastore1: backend1, backend2,..backendN
This is used later to generate the "superdatastore" configuration.

> I'm not sure whether we really need to write a backend. At least for the "tasks
> + events" use case, the Synthesis engine already has a solution. We just need
> to activate it somehow.
Now comes the specific part, the generation of "superdatastore" element is
different case by case (The filtering criteria). At this time I will always
generate a "super calendar" datastore for the calender +event case to work.

> > which will
> > map to (calendar+todo), the format used is vcard2.1 and ical2.0 by default,
> 
> Why vcard2.1?

Because this is the situation in non-joined case, I just don't want to change.

------- Comment #5 From pohly 2009-11-20 01:42:16 PST (-) [reply] -------

(In reply to comment #4)
> (In reply to comment #3)
> > (In reply to comment #2)
> > The solution of this problem should be independent of the Evolution backends.
> > The general task is: "combine two (or more) real data sources into one virtual
> > one".
> Now comes this solution:
> The configuration of type properties is defined as:
> backend1+backend2+...backendN:format1+format2_...formatM (0=<M<=N)

I assume that with "backend" you mean something like "Evolution Contacts". When
that backend gets invoked, how does it know which local database it is supposed
to access?

I suggest to solve this problem by referencing other *source* configurations
instead of *backends*. Then these other source configurations are defined
normally. A sync where a supersource is active is treated like a sync where
each of the involved normal sources is active (database dumps, reporting, ...).

The configuration could look like this:
[calendar+todo]
  type = virtual:text/calendar
  evolutionsource = calendar,todo

It's a bit unfortunate that I followed Evolution's example with the "calendar"
name. Other products use "events", "tasks" for what we call "calendar" and
"todo" and "calendar" for what we have to call "calendar+todo" (just a
proposal, other ideas are welcome).

Note that only data sources with compatible data formats can be merged, as far
as I understand it. Even if the Synthesis engine supports it, I don't think we
have use cases which require it, so we can impose this limitation ourselves and
thus keep the data format selection via [calendar+todo] working as usual.

> The information we lost is: the datastore name is the same as backend name in
> such joined case which I think is OK.
> Another dirty point is the "forced" property in format, the original single
> m_forced property is no longer working.

I think these problems don't occur with the approach above.

> Now comes the specific part, the generation of "superdatastore" element is
> different case by case (The filtering criteria). At this time I will always
> generate a "super calendar" datastore for the calender +event case to work.

Agreed. Later we can add additional configuration parameters which control the
advanced use cases.

------- Comment #6 From Chen Congwu 2009-11-20 02:08:16 PST (-) [reply] -------

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > (In reply to comment #2)
> > > The solution of this problem should be independent of the Evolution backends.
> > > The general task is: "combine two (or more) real data sources into one virtual
> > > one".
> > Now comes this solution:
> > The configuration of type properties is defined as:
> > backend1+backend2+...backendN:format1+format2_...formatM (0=<M<=N)
> 
> I assume that with "backend" you mean something like "Evolution Contacts". When
> that backend gets invoked, how does it know which local database it is supposed
> to access?
That's the part I missed. 
> I suggest to solve this problem by referencing other *source* configurations
> instead of *backends*. Then these other source configurations are defined
> normally. A sync where a supersource is active is treated like a sync where
> each of the involved normal sources is active (database dumps, reporting, ...).
> 
> The configuration could look like this:
> [calendar+todo]
>   type = virtual:text/calendar
>   evolutionsource = calendar,todo
Do you mean add another dummy source folder reference other sources to be
joined ? This simplifies my problem but adding duplicated configuration(in this
case calendar, todo, and calendar+todo are mostly duplicated). This is not very
nice and may confuse user. For example is it meaningful to fire a sync with a
Nokia phone asking two-way sync for calendar and slow-sync for todo? Mostly
this does not work.

> > The information we lost is: the datastore name is the same as backend name in
> > such joined case which I think is OK.
> > Another dirty point is the "forced" property in format, the original single
> > m_forced property is no longer working.
> 
> I think these problems don't occur with the approach above.
I agree with you and actually I already changed my mind during testing.

------- Comment #7 From pohly 2009-11-20 02:23:18 PST (-) [reply] -------

(In reply to comment #6)
> (In reply to comment #5)
> > I suggest to solve this problem by referencing other *source* configurations
> > instead of *backends*. Then these other source configurations are defined
> > normally. A sync where a supersource is active is treated like a sync where
> > each of the involved normal sources is active (database dumps, reporting, ...).
> > 
> > The configuration could look like this:
> > [calendar+todo]
> >   type = virtual:text/calendar
> >   evolutionsource = calendar,todo
> Do you mean add another dummy source folder reference other sources to be
> joined ?

Yes.

> This simplifies my problem but adding duplicated configuration(in this
> case calendar, todo, and calendar+todo are mostly duplicated). This is not very
> nice and may confuse user. For example is it meaningful to fire a sync with a
> Nokia phone asking two-way sync for calendar and slow-sync for todo? Mostly
> this does not work.

[calendar+todo] is indeed mutually exclusive with both [calendar] and [todo].
This is something that users need to know. Depending on how we implement this
"virtual" backend, this semantic can be checked by SyncContext.

In the sync-UI, we can hide the complexity for the default case.

------- Comment #8 From Chen Congwu 2009-11-22 23:21:22 PST (-) [reply] -------

(In reply to comment #7)
> (In reply to comment #6)
> > (In reply to comment #5)
> > > I suggest to solve this problem by referencing other *source* configurations
> > > instead of *backends*. Then these other source configurations are defined
> > > normally. A sync where a supersource is active is treated like a sync where
> > > each of the involved normal sources is active (database dumps, reporting, ...).
> 
Ok, I have pushed the fix to phone branch including several SAN generation
fixes to work with Nokia phone.
I ignore other properties of the virtual syncsource except "type" and
"evolution syncsource" to avoid inconsistency.
The format in the "type" field for a virtual syncsource must be presented,
because this may otherwise leads to inconsistency for different backends
providing different preferred formats.

------- Comment #9 From pohly 2009-11-23 03:15:31 PST (-) [reply] -------

Commit message:
   At this time, the configuration is fixed to work with "calendar+todo" case.

This means "only virtual sync sources which combine one calendar and one todo
are supported, in this order. In other words, a valid configuration is
  evolutionsource=calendar,todo"

Is that correct?

Why was it necessary to change the ISEVENT check in the XML config? A bug in
the original example or in our copy? Please include such information in the
commit message. What I tend to do is to go through a diff while writing the
commit message; that way I notice everything which might require a comment.



+                             "This field is also used to identify a virtual
\n"
+                             "backend which is used to join several sub
backends \n"
+                             "together, Nokia phone typlically needs
todo+calendar \n"
+                             "virtual backends"

Mind the extra white space at line ends. I don't care about those much in
source code, but would like to keep the strings clean. I also suggest a
different text:

=>
-----------------------------
\n
A special "virtual" backend combines several other data sources and presents
them as one set of items to the peer. For example, Nokia phones typically
exchange tasks and events as part of one set of calendar items.\n
\n
Right now such a virtual backend is limited to combining one calendar source
with events and one task source. They have to be specified in the
"evolutionsource" property, typically as "evolutionsource = calendar,todo".\n
\n
In all cases the format of this configuration is
  <backend>[:format]

Different sources combined in one virtual source must have a common
representation. As with other backends, the preferred format can be influenced
via the "format" attribute.
-------------------------------
virtual:text/x-vcalendar - a virtual backend uses vCalendar 1.0 format\n"
=>
virtual:text/x-vcalendar - a virtual backend using vCalendar 1.0 format\n"
-------------------------------

Regarding the new "evolutionsource" parsing: are spaces around a comma allowed?
What if a user has chosen a sync source name which includes a comma? We need
some kind of escaping for this.

std::vector<VirtualSyncSource*> m_virtualDS;

Wouldn't it be better to store shared_ptrs in the vector? The code which uses
plain pointers is legacy code; the Funambol library expected an array of such
pointers.

sourceList.m_virtualDS.push_back (new VirtualSyncSource (params));
would leak the VirtualSyncSource instance if push_back throws an out-of-memory
exception.

+            }
+            else {

The normal formatting is "} else {".

You are not checking whether the virtual sync source is enabled. This has to be
added, because a virtual sync source configured for a SyncML client peer will
show up as inactive source for a SyncML server peer.

SE_THROW ("superdatastore now only supports events+tasks case");

=> the user has no idea what a "superdatastore" is. The error message must
refer to the concepts introduced for the "virtual data source" and include the
source name => vSource->throwError()

------- Comment #10 From Chen Congwu 2009-11-23 04:13:21 PST (-) [reply] -------

(In reply to comment #9)
> Commit message:
>    At this time, the configuration is fixed to work with "calendar+todo" case.

> This means "only virtual sync sources which combine one calendar and one todo
> are supported, in this order. In other words, a valid configuration is
>   evolutionsource=calendar,todo"
> 
> Is that correct?

Correct.

> Why was it necessary to change the ISEVENT check in the XML config? A bug in
> the original example or in our copy? Please include such information in the
> commit message.
That's not a bug. I need to add a field 'KIND' for calendar+todo dispatch, this
filed also covers the original 'ISEVENT field'; therefore I changed it to use
'KIND' instead of 'ISEVENT'.

------- Comment #11 From pohly 2009-11-23 06:33:37 PST (-) [reply] -------

(In reply to comment #10)
> > Why was it necessary to change the ISEVENT check in the XML config? A bug in
> > the original example or in our copy? Please include such information in the
> > commit message.
> That's not a bug. I need to add a field 'KIND' for calendar+todo dispatch, this
> filed also covers the original 'ISEVENT field'; therefore I changed it to use
> 'KIND' instead of 'ISEVENT'.

The upstream libsynthesis syncserv_sample_config.xml still looks wrong to me:
it uses  <dispatchfilter>F.KIND:=EVENT</dispatchfilter> but only defines
ISEVENT  <subprofile name="VTODO" nummandatory="1" showifselectedonly="yes"
field="ISEVENT" value="0">

Your solution solves this and should be reported to Synthesis. For the server
config we don't have a process in place to do this automatically, so an email
is due. Can you notify them?

------- Comment #12 From Chen Congwu 2009-11-24 19:19:59 PST (-) [reply] -------

(In reply to comment #9)
> Commit message:

> Mind the extra white space at line ends. I don't care about those much in
> source code, but would like to keep the strings clean. I also suggest a
> different text:
Updated.

> Regarding the new "evolutionsource" parsing: are spaces around a comma allowed?
> What if a user has chosen a sync source name which includes a comma? We need
> some kind of escaping for this.

Right, I am adding "boost::trim" to strip the spaces and using "\" as the
escape character to handle comma in source name problem. 
Note I am not using SafeConfigNode because it is designed to be used for
internal properties which will is read and written only by SyncEvolution. 
The "evolutionsource" is a public property which maybe set by user explicitly.
Therefore I am just adding escape capable decoding and update the documentation
accordingly.

Pushing to phone branch now.. Please have a look.

------- Comment #13 From pohly 2009-12-18 01:10:34 PST (-) [reply] -------

(In reply to comment #12)
> Pushing to phone branch now..

Merged. I think we have resolved the part we need for SyncEvolution 1.0. For
tracking purposes I am going to split this issue into different parts.



--- Bug imported by patrick.ohly@gmx.de 2012-07-29 20:36 UTC  ---

This bug was previously known as _bug_ 685 at https://bugs.meego.com/show_bug.cgi?id=685