Bug 21466

Some technical notes:

It is not a bug problem to "merge" (virtually, as layout and variants) two symbol files - "normal" and "extra", using rules. So, if we have symbols/ru and symbols/contrib/ru, a small change to the rules would make layout ru(extra1) use symbols/contrib/ru. Well, all these extended variants most probably will have to be mentioned explicitly (or at least generated during the build process).

A bit more tricky is dealing with base.xml. It is monolythic (and, I must admit, I do not think it should be changed). For extending it, some kind of "xml patch" should be used...

I think (do tell if I'm wrong) that we should split this consideration into
end-user/client (accessing layouts and information about them) and
implementation (repository layout, packaging, build) points of view, and
that the two are largely independent, as just about anything can be done at
build time.

To take your example, if the repository/package contains symbols/contrib/ru,
ru(extra1) variant may be made available either by the mechanism you
described, but also by simple concatenation of files at build time, so that
only a composited symbols/ru is installed; and this can later be changed
too. This is the implementation side. What I think is more important to get
right is whether the said variant should really be accessible as
setxkbmap -layout ru -variant extra1 and equivalents. This is the user/
client side.

So I'd first look into the user/client side, and move to implementation
later.

In req. 3 you mention that it would be nice if user could decide whether to
see contrib stuff in the interface or not. But that depends on what we
consider as "interface", the client which the user interacts with. Client
may be setxkbmap, X config file, and a GUI layout switcher may be "dumb"
(not knowing about difference in two categories) or "smart". So we should
decide what to present to each of them through installed layouts and layout
information.

One extreme would be that distinction between categories should be seen
through all clients, and regardless of how smart they are. This means that
the contrib variants would have to have their own installed layouts, e.g. ru
for normal and rucn (*cn for contrib) such that user must do e.g.
setxkbmap -layout rucn -variant extra1 to select the particular contrib
variant; that dumb GUI clients should list e.g. "Russia" and "Russia
(contrib)" layouts, each with their variants; that global information files
(base.xml etc.) should state both categories alike, if contrib was enabled
at build time.

Another extreme would be to cater only for smart GUI clients. In this
scenario, ru may contain all variants alike (from client's POV, regardless
if composited at build or by rules), such that non-GUI switcher/ X config
user has no feeling which variants are normal and which contrib; global
information may be split into two files each (e.g. base.xml and
base-contrib.xml), such that dumb GUI client will neither see nor present to
user contrib variants, and smart can do whatever it wants.

I personally lean toward the former extreme. For certain making sure that
dumb GUI clients will show both categories, and in a most suitable manner.
This implies monolithic installed global info files, with naming such that
indiscriminate parsing and lexical sorting can produce reasonable result in
GUI. Not so sure about separate *cn layouts (so that the difference is felt
even in command line/config files), but they may be induced anyway by
catering for dumb GUI clients. So:

* command line:

      setxkbmap -layout rucn -variant extra1

* config file:

      Option "XkbLayout" "rucn"
      Option "XkbVariant" "extra1"

* dumb GUI client:

      + Russia
        - Legacy
        - Phonetic
        ...
      + Russia (contrib)
        - Extra 1
        - Extra 2
        ...
      + Sweden
        ...
      + Sweden (contrib)
        ...

* smart GUI client:

      + Russia
        - Legacy
        - Phonetic
        ...
      + Sweden
        ...

      [ ] Show contrib layouts
      // ...with whatever fancy GUI presentation when checked.

Finally, the naming "contrib" sounds a bit misleading to me. Both categories
of layouts are obtained through contributions, and both are part of
XKeyboardConfig proper. Instead I'd rather call the category "dedicated" (to
users or certain profession, to some special use case, etc.) or at least
"extras".

> I think (do tell if I'm wrong) that we should split this consideration
I generally agree. But trying to be reasonably lazy, I would like to minimize the changes in the build process;)

> To take your example, if the repository/package contains symbols/contrib/ru,
> ru(extra1) variant may be made available either by the mechanism you
> described, but also by simple concatenation of files at build time, 
I thought about that scenario. But it would make me change the build process seriously. Till that point, the files symbols/* (as well as other files, except rules/*) were used/bundled as they are. With you idea, we'd need some "build area" for concatenated files. I am trying to avoid that, if it is possible.

> too. This is the implementation side. What I think is more important to get
> right is whether the said variant should really be accessible as
> setxkbmap -layout ru -variant extra1 and equivalents. This is the user/
> client side.
Yes, I think that's the right way to go. Also, please keep in mind - we are talking about layouts, not symbols. So even having separate symbol files we could "hide" that fact by using proper rules.

> In req. 3 you mention that it would be nice if user could decide whether to
> see contrib stuff in the interface or not. But that depends on what we
> consider as "interface", the client which the user interacts with.
Well, probably #3 is simply wague. First, xk-c builder should be able to decide whether to provide extras or not. Then, developer of GUI apps should be able to make that decision - show extras or not (if available). And, finally, IF these variants are installed (thanks to the builder), the user of GUI apps could distinguish (if developer provided a way). About the command line - I do not think that distinguishability should be kept. Probably, I would prefer to make that transparent in the command line. What I actually meant is that in the meta-information (base.xml), after merging extras stuff, there is still some way (attribute?) to say "that variant is from extras". That information could be used by GUI developers. But has nothing to do with cmdline.

> Another extreme would be to cater only for smart GUI clients. In this
Yes, probably that's the one I would foresee.

> user has no feeling which variants are normal and which contrib; global
> information may be split into two files each (e.g. base.xml and
> base-contrib.xml), 
Heh.... That's my breaking point. I would struggle for keeping single xml file. But I do not know how to manage xml diff/patch sequence...

> I personally lean toward the former extreme. For certain making sure that
> dumb GUI clients will show both categories, and in a most suitable manner.
Well, since most of the clients are using libxklavier these days (which I have full control over), it is not an issue.

So, my preference would be:

 * command line:
      setxkbmap -layout ru -variant extra1
Regardless whether we have 1 or 2 symbols file. Same for xorg.conf

* dumb GUI client:
      + Russia
        - Legacy
        - Phonetic
        ...
      + Sweden
        ...
* smart GUI client:
      + Russia
        - Legacy
        - Phonetic
        ...
      + Sweden
        ...
      [ ] Show contrib layouts
(only ability to distinguish contrib would differintiate smart from dumb)

> users or certain profession, to some special use case, etc.) or at least
> "extras".
I would see "extras" as preferred word. Let's use it, for now.

> [: Sergey V. Udaltsov :]
> [...]
> Well, since most of the clients are using libxklavier these days (which I
> have full control over) [...] So, my preference would be: [...]

Let's then work from there :)

Firstly, this means that we shouldn't care about clients which don't use
libxklavier to access layout metadata? If so...


> [...]
> there is still some way (attribute?) in [base.xml] to say "that variant is
> from extras".
> [...]
> * dumb GUI client:
> [...]
> * smart GUI client:
> [...]
> (only ability to distinguish contrib would differintiate smart from dumb)

Do I understand correctly that a dumb GUI client should not show extras
variants at all? Under the above assumption, this means that libxklavier
should by default report only normal variants, and its API extended to
provide extras on request too. And then, why not, an attribute seems the
most XML-ish way of stating that a variant is an extra.

(Just as a side note, even if irrelevant: in case there is single metadata
file, with extra variants marked by attribute, a dumb client *not* using
libxklavier would list all variants alike and intermixed, just as it is
now.)

> I generally agree. But trying to be reasonably lazy, I would like to
> minimize the changes in the build process;) [...] I thought about
> [composing single symbols file at build time] scenario. But it would make
> me change the build process seriously. [...] I would struggle for keeping
> single xml file. But I do not know how to manage xml diff/patch
> sequence...

Given that some compositing at build time should anyway be done to create
single metadata files if extras are enabled, I'd just choose whatever
mechanism is more convenient for symbols files as installed (if having two
symbols files for one layout can be made transparent to all target clients,
then ok; if some problems, then compose at build time).

On the other hand, if only smart GUI clients are cared for, which use
libxklavier, which by default hides extras variants (ok, I'm already bulding
upon a lot of assumptions so far :), I wonder why not just install extras
variants unconditionally, without any configure/build calamity, compositing
XML, etc.? I.e. just continue doing as so far, single metadata file in
repository, with the only difference being that added extras variants get
the appropriate attribute.

Anyway, if base.xml is to be composed at build time, why not simply have
base.xml.in as it is now, without extras variants and no special
adjustments, and e.g. base-extras.xml.in with only <layoutList> section such
as:

  <xkbConfigRegistry>

    <modelList></modelList>

    <layoutList>
      ...
      <layout>
        <configItem>
          <name>ru</name>
        </configItem>
        <variantList>
          <variant extra="true">
            <configItem>
              <name>extra1</name>
              <_description>Extra foo bar</_description>
            </configItem>
          </variant>
          ...
        </variantList>
      </layout>
      ...
    </layoutList>

    <optionList></optionList>

  </xkbConfigRegistry>

and write a (Perl?) script to scoop at build time all the extra variants
from base-extras.xml and add them to base.xml, per layout? Providing that
uniform XML coding style is forced as it is now, this script wouldn't even
have to do proper XML parsing, it would easily do its job at line-by-line
level.

> Firstly, this means that we shouldn't care about clients which don't use
> libxklavier to access layout metadata? If so...
Well, yes and no. People still use setxkbmap, and we have to provide for that interface, regardless. But that interface does not have any metadata, so we should not bother about that part. But for apps using base.xml... - I would make reasonably safe bet to say that all apps do it by means of libxklavier. To the best of my knowledge...
 
> Do I understand correctly that a dumb GUI client should not show extras
> variants at all?
You know, at this point, I think that separation "dumb" vs "smart" is unnecessary here. As long as GUI client knows about base.xml (or base-extras.xml, if we introduce it), which is same as "using libxklavier", it is just "GUI client". 

> Under the above assumption, this means that libxklavier
> should by default report only normal variants, and its API extended to
> provide extras on request too. And then, why not, an attribute seems the
> most XML-ish way of stating that a variant is an extra.
I would put it that way: libxklavier always reports all variants. So, if extras are installed - they are reported. But there is an attribute. And then it is up to the application to decide whether to show or hide extra variants (or just ignore that attribute altogether)

> mechanism is more convenient for symbols files as installed (if having two
> symbols files for one layout can be made transparent to all target clients,
> then ok; if some problems, then compose at build time).
There are no problems with having 2 files of symbols. At least I do not see any troubles.

> upon a lot of assumptions so far :), I wonder why not just install extras
> variants unconditionally, without any configure/build calamity, compositing
Some distros might want to cut the slack. Why not give distro builders that control?

> from base-extras.xml and add them to base.xml, per layout? Providing that
> uniform XML coding style is forced as it is now, this script wouldn't even
> have to do proper XML parsing, it would easily do its job at line-by-line
> level.
Well, may be it is an option indeed. Or just tweaking libxklavier a bit, in order to make it use 2 XML trees...

> [: Sergey V. Udaltsov :]
> You know, at this point, I think that separation "dumb" vs "smart" is
> unnecessary here. As long as GUI client knows about base.xml (or base-
> extras.xml, if we introduce it), which is same as "using libxklavier", it
> is just "GUI client".

To me this equates to the target clients being exactly setxkbmap, X config,
and libxklavier. And GUI switchers are then clients to libxklavier, which
indeed means it's irrelevant how "smart" the are, and in fact makes them
entirely irrelevant for this discussion.

> Well, may be it is an option indeed. Or just tweaking libxklavier a bit,
> in order to make it use 2 XML trees...

So, under the assumption that metadata is to always be used through
libxklavier, I too would think it's better that there is no magic whatsoever
at build/install time, that two symbols files per layout are used, and that
the two metadata XML files are handled internally by libxklavier.

(The backward compatibility is kept as well, as older versions of
libxklavier, and hence GUI switchers, would simply not see the extras
variants.)

> To me this equates to the target clients being exactly setxkbmap, X config,
> and libxklavier. And GUI switchers are then clients to libxklavier, which
> indeed means it's irrelevant how "smart" the are, and in fact makes them
> entirely irrelevant for this discussion.
Exactly. The only detail which might distinguish them is "old" (not aware of the new attribute or new xml file) vs "new" (the ones which know about new stuff). BTW, I am still stuck trying to find the good name for the attribute.

> at build/install time, that two symbols files per layout are used, and that
> the two metadata XML files are handled internally by libxklavier.
Yes, it seems so. I will think about it till the next release (which is going to happen in May), then I might start hacking things... This will also give other people time to react, comment, criticize...

> libxklavier, and hence GUI switchers, would simply not see the extras
> variants.)
Yes.

What about per-user settings& Something like ~/.xkeyboard-config

> What about per-user settings& Something like ~/.xkeyboard-config
Could you please elaborate on that? Why xorg.conf (or hal in new xorg) is not good enough for the xkb settings? Or do you mean separate set of symbols/rules/...?

PS Anyway, it is a separate story, I guess. May be worth a separate bug (well, I am a bit of a bureaucrat:)

(In reply to comment #9)
> > What about per-user settings& Something like ~/.xkeyboard-config
> Could you please elaborate on that? Why xorg.conf (or hal in new xorg) is not
> good enough for the xkb settings? Or do you mean separate set of
> symbols/rules/...?
I meant ability to define custom private layout without need to modify system files. Well, it's really another story, so I'll think on it and write other bug after holiday.

(In reply to comment #10)
> (In reply to comment #9)
> > > What about per-user settings& Something like ~/.xkeyboard-config
> > Could you please elaborate on that? Why xorg.conf (or hal in new xorg) is not
> > good enough for the xkb settings? Or do you mean separate set of
> > symbols/rules/...?
> I meant ability to define custom private layout without need to modify system
> files. Well, it's really another story, so I'll think on it and write other bug
> after holiday.

Is this ability handled by kmfl?

http://kmfl.sourceforge.net/

The general structure using base.extras.xml is committed:

http://cgit.freedesktop.org/xkeyboard-config/commit/?id=150850554baeb2d6de124b4c3cc447a694cdc17d

Please test, comment, complain.