Bug 55634

Summary:	TplLogStore implementation for logs retrieved from GMail
Product:	Telepathy	Reporter:	Will Thompson <will>
Component:	logger	Assignee:	Telepathy bugs list <telepathy-bugs>
Status:	RESOLVED MOVED	QA Contact:	Telepathy bugs list <telepathy-bugs>
Severity:	normal
Priority:	medium
Version:	git master
Hardware:	Other
OS:	All
See Also:	https://bugs.freedesktop.org/show_bug.cgi?id=23844
Whiteboard:
i915 platform:		i915 features:
Bug Depends on:	55394
Bug Blocks:
Attachments:	you may want to use a patch like this to test this code

Description Will Thompson 2012-10-04 17:04:01 UTC

I've been working on a TplLogStore implementation which reads mbox files containing logs retrieved from Google (cf. <https://bugzilla.gnome.org/show_bug.cgi?id=594062>).

The branch is still not quite finished: it doesn't implement _tpl_log_store_search_new(), or the new walker API. Also, it's read-only, and wildly inefficient (it re-parses your mail store whenever you call a method on it, basically). But in case anyone is interested, it's sitting at
<http://cgit.collabora.com/git/user/wjt/telepathy-logger/log/?h=gtalk>. The branch looks massive, I accept, but it's under 1500 lines of code, with 700 lines of tests.

----

I am explicitly not solving the problem of how to get the logs from the server onto your local machine. The implementation has a method, _tpl_log_store_gtalk_add_account_mbox(), to map a TpAccount to the path to an mbox file containing the chat logs. The test suite points it at a little mbox file I compiled from my own Google Mail chat logs. The real logger does not, so as it stands it is totally non-functional.

I see a few possible approaches to that problem:

• Add code to guess where the user keeps their IMAP cache, supporting a few popular email clients. Doesn't work for people who only use the GMail web interface for mail, but use Empathy for IM. Also, doesn't scale.
• Require people to configure Evolution for their Google account (via GOA?) and rely on eds syncing their mail in the background.
• Put an IMAP client into the logger, reusing the OAuth 2 token.

Not entirely orthogonally, relying on the entire log being on the disk introduces lag. It might be better to keep a cache on disk, but also ask the server for more logs when something asks the logger for them. (This would probably need new application-facing API; unless it folds into the newfangled log walker.)

Comment 1 Will Thompson 2012-10-05 11:00:44 UTC

(In reply to comment #0)
> The branch is still not quite finished: it doesn't implement
> _tpl_log_store_search_new(), or the new walker API.

It now does implement search_new (but still not the walker API).

I think I'm done with this for a little while.

An interesting refactoring might be to split TplLogStoreGtalkMimeReader in two: one part which turns a GMimeMessage into a GInputStream for its text/xml part, and another one which feeds GInputStreams through the TplLogStoreGtalkReader (the WockyXmppReader subclass for <conversation xmlns=google:conversation> blobs).

This might make it easier to test code in isolation, rather than having to bundle everything up in mbox files, and would make the code better prepared to accept messages via some other path or email library. Given a GInputStream for a conversation, there could be a _pop() method which does:

  while ((stanza = wocky_xmpp_reader_pop_stanza()) == NULL)
    {
      read some input;
      if there's no more input, break;
      else, feed it to the reader;
    }

which would reduce unnecessary parsing of entire conversations (for instance, right now figuring out MUC JIDs involves parsing the entire <conversation>, then just looking at the first <message> within it). I haven't profiled this stuff on actual full-size logs so I don't know if it's necessary. I imagine it's really slow, but…

Comment 2 Will Thompson 2012-10-15 15:10:57 UTC

I've implemented TplLogIter. It's basically a generalization of get_filtered_events() so I refactored things and now get_filtered_events() just uses create_iter(). I think this refactoring may make sense for all stores, not just this one?

Anyway, as this branch stands, merging it would be harmless (modulo the hard dependency on wocky dependency) but useless (because it doesn't know where any of your logs are, it never returns anything). I cooked up a little patch to hard code the path to my own logs, and it works okay—although it's incredibly slow (you can watch the debug output to see it constantly re-scanning the mbox).

Comment 3 Will Thompson 2012-10-15 15:12:52 UTC

Created attachment 68580 [details] [review]
you may want to use a patch like this to test this code


If you want to build this code, you'll need to check out and build Wocky from http://cgit.freedesktop.org/wocky (to $WOCKY_SRCDIR, say) and then run:

PKG_CONFIG_PATH=$WOCKY_SRCDIR/wocky/ ./autogen.sh

so that the logger branch picks up your uninstalled copy of Wocky. (Or you could install it, I guess.)

Comment 4 GitLab Migration User 2019-12-03 19:31:56 UTC

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/telepathy/telepathy-logger/issues/32.

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.