Bug 66618 - watch localed + The SortOrder in SyncEvolution & Evolution-data-server do not match.
Summary: watch localed + The SortOrder in SyncEvolution & Evolution-data-server do no...
Status: RESOLVED FIXED
Alias: None
Product: SyncEvolution
Classification: Unclassified
Component: PIM Manager (show other bugs)
Version: 1.3.99.3
Hardware: Other Linux (All)
: highest major
Assignee: Patrick Ohly
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-07-05 12:41 UTC by Daniel Martin Buckley
Modified: 2014-02-14 11:05 UTC (History)
1 user (show)

See Also:
i915 platform:
i915 features:


Attachments
List of vCards (330.00 KB, application/tar)
2013-07-05 12:41 UTC, Daniel Martin Buckley
Details
vcards for testing (1.03 KB, application/x-gzip)
2013-07-08 13:48 UTC, Patrick Ohly
Details
introduce utility class for monitoring localed (13.26 KB, patch)
2013-07-18 09:05 UTC, Patrick Ohly
Details | Splinter Review
make usage of system D-Bus optional (for testing) (2.18 KB, patch)
2013-07-18 09:06 UTC, Patrick Ohly
Details | Splinter Review
use the helper class to get and adapt locale settings at runtime (17.75 KB, patch)
2013-07-18 09:06 UTC, Patrick Ohly
Details | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Daniel Martin Buckley 2013-07-05 12:41:12 UTC
Created attachment 82073 [details]
List of vCards

When verifying the PinYin support for Search using the UnifiedAddressBook Search in SyncEvolution and the Search in Evolution-Data-Server. It appears that the 
result sort order is not the same.

Step 1) Set locale to zh_CN
Step 2) Create a number of vCards in (English and Chinese)
Step 3) Create a Peer  SetPeer (peer_name_test1, files /htmp/zh2)
Step 4) SyncPeer (peer-name_test1)
Step 5) SetSortOrder(first/last)
Step 6) Search ("")

This returns a list of vCards, the English vCards are correct but the 
Chinese vCards are not in correct order.

My Dictionary

My Dictionary
Chinese                English             Pinyin
鳥                     Bird                  niǎo
犬                     Dog                   quǎn
人                     Man                   rén
人 物                  Person                rén wù
女 性                  Woman                 nǚ xìng
 汽 車                 Car                   qì chē
屋                     House                 wū        
物                     Thing                 wù
五                     Five                   wǔ 
無                     not                    wú
務                     affair                 wù
任 何                  any                    rèn hé
單                     Bill                   dān

VCards and 2 screen shoots attached.
Vcards in  a mixture of English & Chinese
1 screen shot of results side by side
1 screen shot of part of the problem (item 4/5 in list)
Comment 1 Patrick Ohly 2013-07-05 13:30:28 UTC
In the archive you say:

  Entry 4,5 are in reversed order
  鳥        =    niǎo
  女性    =    nǚ xìng

So the question becomes - which order is correct, and which app does not follow it?

According to
http://demo.icu-project.org/icu-bin/locexp?_=zh&d_=en&x=col&collation=pinyin
1. 鳥
2. 女性
is correct.

Daniel, in your report you have two screen shots, but I can't tell which window shows results from EDS and which one from PIM Manager. Can you please clarify?
Comment 2 Daniel Martin Buckley 2013-07-05 15:27:14 UTC
The screenshot is composed of multiple windows on view.

The Center window (containing the field uid: "pas-id-XXXXXXXXXXXXXX" is from
evolution-data-server. The Leftmost window (containing index, fullname, name) 
is from SyncEvolution.
Comment 3 Patrick Ohly 2013-07-05 15:50:55 UTC
(In reply to comment #2)
> The screenshot is composed of multiple windows on view.
> 
> The Center window (containing the field uid: "pas-id-XXXXXXXXXXXXXX" is from
> evolution-data-server. The Leftmost window (containing index, fullname,
> name) 
> is from SyncEvolution.

So it is the PIM Manager which sorts it differently. I'll check.
Comment 4 Daniel Martin Buckley 2013-07-05 15:53:11 UTC
According to
http://demo.icu-project.org/icu-bin/locexp?_=zh&d_=en&x=col&collation=pinyin
1. 鳥
2. 女性
is correct.

Agreed! 

However the output of the SyncEvolution does not match this!

I believe the issue is in SyncEvolution.

Once sorted the correcet order should be this I believe
index 0 = Brian
index 1 = 單  ( dān)
index 2 = Even
index 3 = Evenly
index 4 = 鳥  (niǎo)
index 5 = 女 性 (nǚ xìng)

In SyncEval it is not (index 4 & 5 are reversed)
index 0 = Brian
index 1 = 單  ( dān)
index 2 = Even
index 3 = Evenly
index 4 = 女 性 (nǚ xìng)
index 5 = 鳥  (niǎo)
Comment 5 Patrick Ohly 2013-07-05 16:15:28 UTC
(In reply to comment #4)
> However the output of the SyncEvolution does not match this!
> 
> I believe the issue is in SyncEvolution.

That's what I said - PIM Manager is the part of SyncEvolution which does the sorting ;-)
Comment 6 Patrick Ohly 2013-07-08 13:27:29 UTC
I noticed that at least one of the contact files is invalid:

BEGIN:VCARD
VERSION:3.0
FN:Brian
N:Brian;;;;
TYPE=WORK,FAX:003538712310010
ORG:18-Brian
END:VCARD

A "TEL:" is missing in the TYPE=WORK,FAX:003538712310010 line.

Several other files had extra white spaces at the end of a line or before/after the vcard.

This has nothing to do with the actual problem, I just wonder how the files were created.
Comment 7 Daniel Martin Buckley 2013-07-08 13:31:57 UTC
These files was manually created using VI on linux for test purposes.

They were "Sync" via the the file import (not via bluetooth)

I was not testing the Telephone No. feature so I was not worried about the phone no. but thanks for catching it. It will save me time when I come to testing the telephone no.
Comment 8 Patrick Ohly 2013-07-08 13:48:31 UTC
Created attachment 82186 [details]
vcards for testing

I've not been able to reproduce this issue.

Here's what I have done:

$ syncevolution --daemon=no --delete-items backend=evolution-contacts database='' --luids \*
$ syncevolution --daemon=no --import /tmp/zh2 backend=evolution-contacts database='' 
#0: 1.vcf: pas-id-51DAC11400000065
#1: 10.vcf: pas-id-51DAC11400000066
#2: 11.vcf: pas-id-51DAC11400000067
#3: 12.vcf: pas-id-51DAC11400000068
#4: 13.vcf: pas-id-51DAC11400000069
#5: 14.vcf: pas-id-51DAC1140000006A
#6: 15.vcf: pas-id-51DAC1140000006B
#7: 16.vcf: pas-id-51DAC1140000006C
#8: 17.vcf: pas-id-51DAC1140000006D
#9: 18.vcf: pas-id-51DAC1140000006E
#10: 19.vcf: pas-id-51DAC1140000006F
#11: 2.vcf: pas-id-51DAC11400000070
#12: 20.vcf: pas-id-51DAC11400000071
#13: 3.vcf: pas-id-51DAC11400000072
#14: 4.vcf: pas-id-51DAC11400000073
#15: 5.vcf: pas-id-51DAC11400000074
#16: 6.vcf: pas-id-51DAC11400000075
#17: 7.vcf: pas-id-51DAC11400000076
#18: 8.vcf: pas-id-51DAC11400000077
#19: 9.vcf: pas-id-51DAC11400000078
$ LC_TYPE=zh_CN.UTF-8 LC_ALL=zh_CN.UTF-8 LANG=zh_CN.UTF-8 syncevo-dbus-server &
[3] 15550
$ ~pohly/src/syncevolution/src/dbus/server/pim/examples/search.py -s '[]' -a '' -r -m -o first/last
...

=> 
* 000 Brian
     {'roles': [{'organisation': '18-Brian'}], 'phones': [('TYPE=WORK,FAX:003538712310010', [])], 'full-name': 'Brian', 'source': [('', 'pas-id-51DAC1140000006E')], 'structured-name': {'family': 'Brian'}, 'id': 'd6e7f928afeffa05a9061e52d7bb6037cf031744'}

* 001 單
     {'full-name': u'\u55ae', 'source': [('', 'pas-id-51DAC11400000069')], 'structured-name': {'family': u'\u55ae'}, 'id': '981377f4e771e5c27533511777987c2597330ed7', 'roles': [{'organisation': u'13-d\u0101n'}]}

* 002 Evan
     {'roles': [{'organisation': '19-Evan'}], 'phones': [('+3955510010', ['home', 'voice'])], 'full-name': 'Evan', 'source': [('', 'pas-id-51DAC1140000006F')], 'structured-name': {'family': 'Evan'}, 'id': 'd221126a2591ccbb8327dd7da2d8557598dcc24d'}

* 003 Evanly
     {'roles': [{'organisation': '20-Evan'}], 'phones': [('(111) 555-10010', ['voice', 'work'])], 'full-name': 'Evanly', 'source': [('', 'pas-id-51DAC11400000071')], 'structured-name': {'family': 'Evanly'}, 'id': '1d9076397a9052264694385718d0915203d24aba'}

* 004 鳥
     {'roles': [{'organisation': u'01-ni\u01ceo'}], 'phones': [(' +353 214343730', ['voice', 'work'])], 'full-name': u'\u9ce5', 'source': [('', 'pas-id-51DAC11400000065')], 'structured-name': {'family': u'\u9ce5'}, 'id': '247bc737fa5b823f039e94470e797587af8bb39f'}

* 005 女性
     {'full-name': u'\u5973\u6027', 'source': [('', 'pas-id-51DAC11400000074')], 'structured-name': {'family': u'\u5973\u6027'}, 'id': '618fe5523a3c74426509233843a17b586ea01f05', 'roles': [{'organisation': u'05-n\u01da x\xecng'}]}

* 006 汽車
     {'full-name': u'\u6c7d\u8eca', 'source': [('', 'pas-id-51DAC11400000075')], 'structured-name': {'family': u'\u6c7d\u8eca'}, 'id': 'c298e80abd5109d4d0db296b9df3430e823d956d', 'roles': [{'organisation': u'06-q\xec ch\u0113'}]}

* 007 犬
     {'roles': [{'organisation': u'02-qu\u01cen'}], 'phones': [(' 00353 214343731', ['home', 'voice'])], 'full-name': u'\u72ac', 'source': [('', 'pas-id-51DAC11400000070')], 'structured-name': {'family': u'\u72ac'}, 'id': '34d2594b50b8800664b808db802ef91f33e57f83'}

* 008 人
     {'roles': [{'organisation': u'03-r\xe9n'}], 'phones': [(' (021) 4343733', ['voice', 'work'])], 'full-name': u'\u4eba', 'source': [('', 'pas-id-51DAC11400000072')], 'structured-name': {'family': u'\u4eba'}, 'id': '505ca0dfc995ee408d68a71ff43c8433f22f6eb2'}

* 009 任何
     {'full-name': u'\u4efb\u4f55', 'source': [('', 'pas-id-51DAC11400000068')], 'structured-name': {'family': u'\u4efb\u4f55'}, 'id': 'fd358214c039a03c8b87f04162825a983998d395', 'roles': [{'organisation': u'12-r\xe8n h\xe9'}]}

* 010 人物
     {'roles': [{'organisation': u'04-r\xe9n w\xf9'}], 'phones': [(' 0214343734', ['voice', 'work'])], 'full-name': u'\u4eba\u7269', 'source': [('', 'pas-id-51DAC11400000073')], 'structured-name': {'family': u'\u4eba\u7269'}, 'id': '9d411a655d3df4193ee19a239b06ded4fa1af59b'}

* 011 Tom
     {'full-name': 'Tom', 'source': [('', 'pas-id-51DAC1140000006A')], 'structured-name': {'family': 'Tom'}, 'id': '53c08b49402fb8a362346ce5b4b7be7baddba138', 'roles': [{'organisation': '14-Tom'}]}

* 012 Will
     {'full-name': 'Will', 'source': [('', 'pas-id-51DAC1140000006C')], 'structured-name': {'family': 'Will'}, 'id': '29b20705bffe3411345962f368ea2a3d1c0d18c1', 'roles': [{'organisation': '16-Will'}]}

* 013 William
     {'full-name': 'William', 'source': [('', 'pas-id-51DAC1140000006B')], 'structured-name': {'family': 'William'}, 'id': 'd58628fd36f114205f53fe410da0e2e39903abf3', 'roles': [{'organisation': '15-William'}]}

* 014 屋
     {'full-name': u'\u5c4b', 'source': [('', 'pas-id-51DAC11400000076')], 'structured-name': {'family': u'\u5c4b'}, 'id': 'eebd8b9d701217a878b78c01a68f384366a63a1d', 'roles': [{'organisation': u'07-w\u016b'}]}

* 015 無
     {'full-name': u'\u7121', 'source': [('', 'pas-id-51DAC11400000066')], 'structured-name': {'family': u'\u7121'}, 'id': '0c5e6b42b48e7dd0a09b1d40f7811740b9732ec4', 'roles': [{'organisation': u'10-w\xfa'}]}

* 016 五
     {'full-name': u'\u4e94', 'source': [('', 'pas-id-51DAC11400000078')], 'structured-name': {'family': u'\u4e94'}, 'id': '7984f78af60b238fedea5921f5902f7a2312ee90', 'roles': [{'organisation': '09-wu'}]}

* 017 物
     {'full-name': u'\u7269', 'source': [('', 'pas-id-51DAC11400000077')], 'structured-name': {'family': u'\u7269'}, 'id': '141631f4f690ba8818f976401891703842dae122', 'roles': [{'organisation': u'08-w\xf9'}]}

* 018 務
     {'full-name': u'\u52d9', 'source': [('', 'pas-id-51DAC11400000067')], 'structured-name': {'family': u'\u52d9'}, 'id': '51ae00c3ea14e3bd9cd02765467d4d55f4a37ded', 'roles': [{'organisation': u'11-w\xf9'}]}

* 019 Wylam
     {'full-name': 'Wylam', 'source': [('', 'pas-id-51DAC1140000006D')], 'structured-name': {'family': 'Wylam'}, 'id': 'b03593dc24ec111703203a698a08cc135bfc3c10', 'roles': [{'organisation': '17-Wylam'}]}

Note that 鳥 comes before 女性 at index 4 resp. 5, as expected.

Attached the cleaned up vcards that I was using.

Which version of SyncEvolution were you using? See "syncevolution --version".

Was the locale set in the environment before starting syncevo-dbus-server? See "strings /proc/`pidof syncevo-dbus-server`/environ | grep zh_CN".

Can you try to reproduce it exactly as above? The search.py script is part of the source tree and distribution.
Comment 9 Daniel Martin Buckley 2013-07-08 14:48:08 UTC
Which version of SyncEvolution were you using? See "syncevolution --version".

Syncevolution Version

SyncEvolution 1.3.99.3+20130529+SE+b989f69+SYSYNC+3366831

-----------------------------------------------------------
Was the locale set in the environment before starting syncevo-dbus-server? See
"strings /proc/`pidof syncevo-dbus-server`/environ | grep zh_CN".

Syncevo-dbus-server is started automatically on bootup!

I do not have the system at the moment ..so I will back to you with desired information!

-----------------------------------------------------------
The method used to create the files was as follows

on default locale (POSIX)
1) Created the vcards. (In fact they were copied from another machine to save time)
2) SetPeer test1 files /tmp/zh2
3) SyncPeer test1
4) Listed all the pears via device view
5) List all the pears in Unified Address Book
6) Change Locale zh_CN & reboot 
7) Listed all the pears via device view
8) List all the pears in Unified Address Book
 
-----------------------------------------------------------
I will install the search.py and any additional files I need to get your requested information.
Comment 10 Patrick Ohly 2013-07-08 16:35:58 UTC
> I will install the search.py and any additional files I need to get your
> requested information.

The fist check should be for the environment of the running syncevo-dbus-server. If it doesn't get the right env variables when it starts, then it will not adapt to changes of those settings later on. Adding that feature is on my list, but wasn't part of the original plan (and delivery). If this issue here turns out to be caused by the lack of runtime adjustment to locale changes, then we can use it to track the feature request.
Comment 11 Daniel Martin Buckley 2013-07-09 12:10:24 UTC
Sorry for the delay!

In answer to your question :- 

Was the locale set in the environment before starting syncevo-dbus-server? See "strings /proc/`pidof syncevo-dbus-server`/environ | grep zh_CN".

No locale was set in /etc/***/environ
Comment 12 Patrick Ohly 2013-07-09 12:58:27 UTC
(In reply to comment #11)
> Sorry for the delay!
> 
> In answer to your question :- 
> 
> Was the locale set in the environment before starting syncevo-dbus-server?
> See "strings /proc/`pidof syncevo-dbus-server`/environ | grep zh_CN".
> 
> No locale was set in /etc/***/environ

That doesn't quite answer the question. Even if nothing is set in /etc, the process might still get spawned with the right environment variables. To be absolutely sure, please run the "strings" command above.
Comment 13 Daniel Martin Buckley 2013-07-09 14:02:50 UTC
Unable to run exact command .. no pidof

So manually

ps aux | grep syncevo-dbus

root 2717 0.0 1.5 119312 16468 ? S1 14:18 0:00 /usr/libexec/syncevo-dbus-server


strings /proc/2717/environ | grep zh_CN

nothing!!!

strings /proc/2717/environ | grep en_GB

nothing!!!
Comment 14 Patrick Ohly 2013-07-09 14:13:13 UTC
(In reply to comment #13)
> Unable to run exact command .. no pidof
> 
> So manually
> 
> ps aux | grep syncevo-dbus
> 
> root 2717 0.0 1.5 119312 16468 ? S1 14:18 0:00
> /usr/libexec/syncevo-dbus-server
> 
> 
> strings /proc/2717/environ | grep zh_CN
> 
> nothing!!!
> 
> strings /proc/2717/environ | grep en_GB
> 
> nothing!!!

That probably explains the problem. Can you double check that running with valid env variable makes the sorting work as expected? Kill the process above, then run it with DBUS_SESSION_BUS_ADDRESS and LC_* variable set.

You should also fix your system such that these env variables get set in the environment of all process before they start. For SyncEvolution and in particular with EDS you really want to avoid the situation where it starts with unset env variables, then changes at runtime to the real ones. In EDS, that can lead to rewriting the database twice (go to C locale, back to real one).
Comment 15 Patrick Ohly 2013-07-18 09:05:35 UTC
Created attachment 82578 [details] [review]
introduce utility class for monitoring localed
Comment 16 Patrick Ohly 2013-07-18 09:06:07 UTC
Created attachment 82579 [details] [review]
make usage of system D-Bus optional (for testing)
Comment 17 Patrick Ohly 2013-07-18 09:06:51 UTC
Created attachment 82580 [details] [review]
use the helper class to get and adapt locale settings at runtime
Comment 18 Patrick Ohly 2014-02-14 10:24:52 UTC
Daniel, what is the status with this issue? Support for localed has been in SyncEvolution for a while. Is that good enough so that we can close this issue?
Comment 19 Daniel Martin Buckley 2014-02-14 10:45:48 UTC
I believe this is now working!

It can be set to closed!


Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.