Bug 24206

I've been working through filling in README's with a basic form that
gives pointers to git repos, bugzilla, the mailing lists and wiki,
since getting feedback a while ago that for people who didn't already
know which modules came from X.Org, they had no idea where to find 
updates, report bugs, or request help.   I should probably finish 
running my script on the rest of the modules and then push the README's
so at least they're all in git.

COPYING updates are also a work in progress - there's still far too many
with the placeholder stub file.

Revised suggestions:

1) Add the foreign option to those components missing it
2) Delete all zero byte NEWS, README and AUTHORS file. The content of existing files are to be reviewed as per existing process, not the subject of this bug
3) INSTALL: rename the 2 hand written ones and generate the default for all components

I have prototyped the generation of the INSTALL file. To summarize:

In util-macros:
Check-in the sole copy of INSTALL in git
Add "dist_doc_DATA = INSTALL" to the makefile to install in share/doc/util-macros
Add an XORG_INSTALL macro which copies the file from doc to component distdir

In all components:
Add XORG_INSTALL (preferably XORG_DEFAULT_OPTIONS) to configure.ac
Update makefile.am:
DISTCLEANFILES = $(PACKAGE)-$(VERSION).tar.* ChangeLog INSTALL
.PHONY: ChangeLog INSTALL
ChangeLog:
	$(CHANGELOG_CMD)
INSTALL:
	$(INSTALL_CMD)

I have tested with dist, distcheck, distclean and distcleancheck on a VPATH build. The override of distcleancheck_listfiles is not required.

(In reply to comment #2)
> Revised suggestions:
> 
> 1) Add the foreign option to those components missing it

ACK

> 2) Delete all zero byte NEWS, README and AUTHORS file. The content of existing
> files are to be reviewed as per existing process, not the subject of this bug

Please synchronise this with Alan. As he said, he already did many READMEs, so it'd best if you could do the deletion at the same time of the replacement default README.

> 3) INSTALL: rename the 2 hand written ones and generate the default for all
> components
> 
> I have prototyped the generation of the INSTALL file. To summarize:
> 
> In util-macros:
> Check-in the sole copy of INSTALL in git

The two problems I see with that:
- anyone who clones a directory will not see an INSTALL telling them what to do until they also clone the macros. At which point they've already found some other source for installation instructions and the point of having an INSTALL is moot. However, looking at e.g. xclock's INSTALL, it is geared at the tarball only anyway and mostly useless for git clones. So as long as we're happy with this status quo, fine.
- if a component needs a custom INSTALL file, how will you handle this?

(In reply to comment #3)
> (In reply to comment #2)
> > Revised suggestions:
> > 
> > 1) Add the foreign option to those components missing it
> 
> ACK
> 
> > 2) Delete all zero byte NEWS, README and AUTHORS file. The content of existing
> > files are to be reviewed as per existing process, not the subject of this bug
> 
> Please synchronise this with Alan. As he said, he already did many READMEs, so
> it'd best if you could do the deletion at the same time of the replacement
> default README.

I had that in mind.
> 
> > 3) INSTALL: rename the 2 hand written ones and generate the default for all
> > components
> > 
> > I have prototyped the generation of the INSTALL file. To summarize:
> > 
> > In util-macros:
> > Check-in the sole copy of INSTALL in git
> 
> The two problems I see with that:
> - anyone who clones a directory will not see an INSTALL telling them what to do
> until they also clone the macros. At which point they've already found some
> other source for installation instructions and the point of having an INSTALL
> is moot. However, looking at e.g. xclock's INSTALL, it is geared at the tarball

That was my assumption, tarball. I came from the wiki to get into git, but I don't know about others. Can't help for that one.

> only anyway and mostly useless for git clones. So as long as we're happy with
> this status quo, fine.
> - if a component needs a custom INSTALL file, how will you handle this?
> 

There are 2 scenarios in your question:

1) The tarball installation requires special instructions (e.g options to configure script). 

It would be best to copy and modify the INSTALL file, removing the mechanics in the makefile and not generate it. Check-in and update .gitignore.

2) Additional instructions, supplementing, but unrelated to the tarball installation. For example the 2 video hand written INSTALL files also talk about how to configure the video driver once the tarball is installed.

The generated file is still applicable, so I would add a new file, perhaps configure.txt or something, with the extra information.

Each component is independent, so we have the flexibility.

I pushed READMEs for most of the app/* modules last week, so there should be
many fewer 0-byte ones now.

Created attachment 30293 [details] [review]
[PATCH] Add an XORG_INSTALL macro to allow comps to copy the INSTALL file

Require patch 0004 from 24450 for other comps to use this macro

Created attachment 30294 [details] [review]
[PATCH] Bump to 1.4.0 due to new XORG_INSTALL

May require patch 0002 from 24450 due to configure.ac.

Created attachment 30463 [details] [review]
 [PATCH] Install the INSTALL file in share/doc/util-macros

This is the unique location where other components can copy the INSTALL file from the util-macros modules.

On Mon, Sep 28, 2009 at 17:30:35 -0700, bugzilla-daemon@freedesktop.org wrote:

> 6) INSTALL: it can be generated using the ChangeLog design. Delete from git.
> 
can't we just keep relying on automake to put INSTALL in the tarballs,
rather than keep some version of it in whatever repo?

foreign should now be in the modules that don't have it.  I don't have permission to push to xkeyboard-config, so that module is still lacking.

(In reply to comment #9)
> On Mon, Sep 28, 2009 at 17:30:35 -0700, bugzilla-daemon@freedesktop.org wrote:
> 
> > 6) INSTALL: it can be generated using the ChangeLog design. Delete from git.
> > 
> can't we just keep relying on automake to put INSTALL in the tarballs,
> rather than keep some version of it in whatever repo?
> 

This is how it works for GNU projects (strictness=gnu). It's one of the rules that Automake enforces for GNU projects. There are other rules that it also enforces. X.Org project is not a GNU software, so we must use the "foreign" strictness. I wish the rules automake enforces be selectable, so we could choose to obtain the INSTALL file. Unfortunately they have chosen "policy" over "mechanism".

Quote from automake doc:
The valid strictness levels are:

foreign
    Automake will check for only those things that are absolutely required for proper operations. For instance, whereas GNU standards dictate the existence of a NEWS file, it will not be required in this mode. The name comes from the fact that Automake is intended to be used for GNU programs; these relaxed rules are not the standard mode of operation.

gnu
    Automake will check—as much as possible—for compliance to the GNU standards for packages. This is the default.

Summary of changes
-------------------

XORG_INSTALL macro is contained in XORG_DEFAULT_OPTIONS. It copies the INSTALL file (in util-macros git repo) to $prefix/share/doc/util-macros to the target module directory.

All git INSTALL files are deleted (save for util-macros)
For driver input-synaptic and video-radeonhd, the hand written INSTALL has been moved to driver.txt in the share/doc directory like other modules do. I'll implement another solution if need be.

All zero byte README, NEWS, AUTHORS are deleted from git

All modules have AM_INIT_AUTOMAKE([foreign dist-bzip2])

In proto and font, README and COPYING have been reviewed/fixed

As of today, left to do:
The remainder modules for REAME and COPYING 

(In reply to comment #12)
> Summary of changes
> -------------------
> 
> XORG_INSTALL macro is contained in XORG_DEFAULT_OPTIONS. It copies the INSTALL
> file (in util-macros git repo) to $prefix/share/doc/util-macros to the target
> module directory.
> 
> All git INSTALL files are deleted (save for util-macros)
> For driver input-synaptic and video-radeonhd, the hand written INSTALL has been
> moved to driver.txt in the share/doc directory like other modules do. I'll
> implement another solution if need be.
> 
> All zero byte README, NEWS, AUTHORS are deleted from git
> 
> All modules have AM_INIT_AUTOMAKE([foreign dist-bzip2])
> 
> In proto and font, README and COPYING have been reviewed/fixed
> 
> As of today, left to do:
> The remainder modules for REAME and COPYING 
> 

Patches have been created for all 251 xorg/* modules but not released yet. The .gitignore file will ignore the INSTALL file

All patches applied.

These recent changes make 'distcheck' fail because of a missing INSTALL file:

checking whether XSERVER_LIBPCIACCESS is declared... yes
checking for ANSI C header files... (cached) yes
checking for PCIACCESS... yes
checking whether _LP64 is declared... no
checking whether __amd64__ is declared... no
checking whether amd64 is declared... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
(GIT_DIR=./.git git log > .changelog.tmp && mv .changelog.tmp
ChangeLog) || (rm -f .changelog.tmp; touch ChangeLog; echo 'git
directory not found: installing possibly empty changelog.' >&2)
{ test ! -d "xf86-video-geode-2.11.6" || { find
"xf86-video-geode-2.11.6" -type d ! -perm -200 -exec chmod u+w {} ';'
&& rm -fr "xf86-video-geode-2.11.6"; }; }
test -d "xf86-video-geode-2.11.6" || mkdir "xf86-video-geode-2.11.6"
cp: cannot stat `./INSTALL': No such file or directory
make: *** [distdir] Error 1

Previously, running "./autogen.sh && make distcheck" worked just fine. This is no longer the case.

(In reply to comment #15)
> These recent changes make 'distcheck' fail because of a missing INSTALL file:
> 
> checking whether XSERVER_LIBPCIACCESS is declared... yes
> checking for ANSI C header files... (cached) yes
> checking for PCIACCESS... yes
> checking whether _LP64 is declared... no
> checking whether __amd64__ is declared... no
> checking whether amd64 is declared... no
> configure: creating ./config.status
> config.status: creating Makefile
> config.status: creating src/Makefile
> config.status: creating config.h
> config.status: executing depfiles commands
> config.status: executing libtool commands
> (GIT_DIR=./.git git log > .changelog.tmp && mv .changelog.tmp
> ChangeLog) || (rm -f .changelog.tmp; touch ChangeLog; echo 'git
> directory not found: installing possibly empty changelog.' >&2)
> { test ! -d "xf86-video-geode-2.11.6" || { find
> "xf86-video-geode-2.11.6" -type d ! -perm -200 -exec chmod u+w {} ';'
> && rm -fr "xf86-video-geode-2.11.6"; }; }
> test -d "xf86-video-geode-2.11.6" || mkdir "xf86-video-geode-2.11.6"
> cp: cannot stat `./INSTALL': No such file or directory
> make: *** [distdir] Error 1
> 
> Previously, running "./autogen.sh && make distcheck" worked just fine. This is
> no longer the case.
> 

This was fixed by a new version of XORG_INSTALL macro in util/macros
The correct version is:
AC_REQUIRE([PKG_PROG_PKG_CONFIG])
macros_docdir=`$PKG_CONFIG --print-errors --variable=docdir xorg-macros`
INSTALL_CMD="(cp -f "$macros_docdir/INSTALL" \$(top_srcdir)/.INSTALL.tmp && \
mv \$(top_srcdir)/.INSTALL.tmp \$(top_srcdir)/INSTALL) \
|| (rm -f \$(top_srcdir)/.INSTALL.tmp; touch \$(top_srcdir)/INSTALL; \
echo 'util-macros \"docdir\" from xorg-macros.pc not found: installing possibly empty INSTALL.' >&2)"
AC_SUBST([INSTALL_CMD])

Since then, the macros version has been bumped to 1.4.1. Also check that the search path of the PKG_CONFIG* variables does not pick-up an older version. The macros are now installed in "share/pkgconfig", the arch independent location.

Has it occurred to anyone that mass-upgrading drivers to use a new macro that has not yet been packaged and released by all major distributions is a bad idea?

No.  That is not a problem.

We don't wait for downstream packagers to pull in something before we use it in our codebase.  That's absurd.

If you're using git as the source for your drivers, then you should be comfortable using util/macros from git as well...

If you're not comfortable with that, then you should wait for drivers to be released by your distribution.

> --- Comment #16 from Gaetan Nadon <memsize@videotron.ca>  2009-12-26 09:32:08 PST ---
> Since then, the macros version has been bumped to 1.4.1. Also check that the
> search path of the PKG_CONFIG* variables does not pick-up an older version. The
> macros are now installed in "share/pkgconfig", the arch independent location.
> 
Gaetan, bumping the macros version (and releasing) should have been done
before changing other modules to require the new version.  and the
macros requirement for other modules should have been bumped in
configure.ac to make this dependency obvious, instead of silently
breaking distcheck.  just in case you (or somebody else) want to do
something like that again in the future.

I am delighted we are having this discussion. I have been trying to figure out the right course of action in this area. 

By default, (past projects policies) I agree with Jeremy, one should update the build regularly to avoid breakage due to code changes in dependencies. It can happen at any time from anywhere. The git master branch is work in progress.

When observing the previous version bumps in macros, it's about once per release. This confirms Jeremy's position. The recent addition of XORG_INSTALL is consumed by all 240 modules, so it's 240 patches to create and push. Not too bad if you do it once every 6 months.

What complicated things a bit more is that there has been a "mini" release recently where a few modules have been published. Some of them required a more recent version of macros, hence the bump to 1.4.1 which I was not aware of. The xserver was the one depending on new version of macros, but configure.ac was not updated to require 1.4.

Some factors to consider
------------------------

* Project wide macros (240+ modules) are not the norm, won't happen often
* It would be an insane amount of work to update version number of macros or any other modules, every time it introduces a breakage.
* There is nothing wrong in occasionally bumping a version number mid-release under exceptional circumstances.

I am considering a patch for each module to bump required macros to 1.4 from 1.3 and AC_PREREQ to 2.60 from 2.57 (which I was planing to do anyway). If this can save time and aggravation, then it is worthed.

The standard behavior for the future is to bump by one for every release (if need be) and adjust modules configure.ac if necessary. Anything else would be by "popular demand".

Be prepared to have some people complain about the new 1.4 requirement in their module. They want to us the last system shipped 1.3 with the new code in git.

What do you think?




 

> --- Comment #20 from Gaetan Nadon <memsize@videotron.ca>  2009-12-26 11:50:22 PST ---
> Be prepared to have some people complain about the new 1.4 requirement in their
> module. They want to us the last system shipped 1.3 with the new code in git.
> 
The requirement is already there.  You added it when making modules
use $(INSTALL_CMD) in Makefile.am.  All I'm asking is for that
requirement to be made explicit, rather than introducing silent
breakage.

I really don't like the idea of bumping AC_PREREQ to 2.60, at a time when neither of the distros I develop for have it packaged, for the same reasons that I loathe this intrusive change to the xf86-video-geode's macros. 

In case this wasn't obvious, even developers use normal distributions as their development environment and none of those are called Moving Target GIT/Linux.

> --- Comment #22 from Martin-Éric Racine <q-funk@iki.fi>  2009-12-26 12:11:29 PST ---
> I really don't like the idea of bumping AC_PREREQ to 2.60, at a time when
> neither of the distros I develop for have it packaged

This doesn't make sense, autoconf 2.60 is 3 and a half years old.

(In reply to comment #22)
> I really don't like the idea of bumping AC_PREREQ to 2.60, at a time when
> neither of the distros I develop for have it packaged, for the same reasons
> that I loathe this intrusive change to the xf86-video-geode's macros. 

This is very interesting. I posted on the list a while ago on that topic and got no response. It's a different issue, of course. Would you be kind enough to e-mail me distro/OS commonly used that don't have 2.60? Version 2.57 is 8 years old. We need to find a happy medium. Can't explain it all in this bug report.

> 
> In case this wasn't obvious, even developers use normal distributions as their
> development environment and none of those are called Moving Target GIT/Linux.
> 

Nope, it wasn't obvious. I figured it out a while ago when someone complained about changes in git that did not work against last release packages. I understand for xserver or libs that must maintain backward comparability, but for old build tools to build new code, no.

The proposed changes always get reviewed on the list. This is where we discuss the merit of the changes and their impact. 

I appreciate all the comments.

For various reasons, if we need to bump AC_PREREQ, I'd go for 2.59, which seems to be widespread enough to satisfy the needs of even the most deprecated distros.

On Sat, Dec 26, 2009 at 11:57:08PM -0800, bugzilla-daemon@freedesktop.org wrote:
> --- Comment #25 from Martin-Éric Racine <q-funk@iki.fi>  2009-12-26 23:57:07 PST ---
> For various reasons, if we need to bump AC_PREREQ, I'd go for 2.59, which seems
> to be widespread enough to satisfy the needs of even the most deprecated
> distros.

Since Debian oldstable (aka etch) has 2.60, I assume you're talking
about RHEL5, which ships 2.59.  Are you building these drivers on RHEL
or CentOS?

(In reply to comment #26)
> On Sat, Dec 26, 2009 at 11:57:08PM -0800, bugzilla-daemon@freedesktop.org
> wrote:
> > --- Comment #25 from Martin-Éric Racine <q-funk@iki.fi>  2009-12-26 23:57:07 PST ---
> > For various reasons, if we need to bump AC_PREREQ, I'd go for 2.59, which seems
> > to be widespread enough to satisfy the needs of even the most deprecated
> > distros.
> 
> Since Debian oldstable (aka etch) has 2.60, I assume you're talking
> about RHEL5, which ships 2.59.  Are you building these drivers on RHEL
> or CentOS?
> 

I just tried geode under 2.59 and it fails to configure (it isn't my doing). All xorg modules configure under 2.60. There is a 3 year gap between 2.59 and 2.60. On OpenSolaris, they have 2.61 which same year as 2.60 (2006). 

I think geode is a driver for an embedded system and has a pre-assembled dev environment. In any case, there are always 2 options: upgrade the tool chain or write a configure.ac to provide the same functions on a back-level autoconf. There is a limit however, the xorg required minimum level of automake is 1.9 which requires autoconf 2.58. The module is standing right on the edge.

I can help with special situations if I am provided with the necessary information, but there are project wide requirements that need to go forward. 

(In reply to comment #20)
> When observing the previous version bumps in macros, it's about once per
> release. This confirms Jeremy's position. The recent addition of XORG_INSTALL
> is consumed by all 240 modules, so it's 240 patches to create and push. Not too
> bad if you do it once every 6 months.
> 
> What complicated things a bit more is that there has been a "mini" release
> recently where a few modules have been published. Some of them required a more
> recent version of macros, hence the bump to 1.4.1 which I was not aware of. The
> xserver was the one depending on new version of macros, but configure.ac was
> not updated to require 1.4.

Xorg modules can be and are released at any time.   While there's often
a flurry around katamari releases, if you browse the xorg-announce archives,
you'll see there's a stream of releases year round, as individual modules
are ready at different times (drivers when new hardware is ready or a new
server ABI is published, applications when there's new features or I have
a few spare minutes to push out yet another release of one of the 
non-katamari apps that need them.)

The next planned katamari release is about 9 months away, but dozens of
modules will be released individually before then - especially with an
ABI-breaking Xorg server 1.8 release planned in about 3 months.

(In reply to comment #28)
> (In reply to comment #20)
> > When observing the previous version bumps in macros, it's about once per
> > release. This confirms Jeremy's position. The recent addition of XORG_INSTALL
> > is consumed by all 240 modules, so it's 240 patches to create and push. Not too
> > bad if you do it once every 6 months.
> > 
> > What complicated things a bit more is that there has been a "mini" release
> > recently where a few modules have been published. Some of them required a more
> > recent version of macros, hence the bump to 1.4.1 which I was not aware of. The
> > xserver was the one depending on new version of macros, but configure.ac was
> > not updated to require 1.4.
> 
> Xorg modules can be and are released at any time.   While there's often
> a flurry around katamari releases, if you browse the xorg-announce archives,
> you'll see there's a stream of releases year round, as individual modules
> are ready at different times (drivers when new hardware is ready or a new
> server ABI is published, applications when there's new features or I have
> a few spare minutes to push out yet another release of one of the 
> non-katamari apps that need them.)
> 
> The next planned katamari release is about 9 months away, but dozens of
> modules will be released individually before then - especially with an
> ABI-breaking Xorg server 1.8 release planned in about 3 months.
> 

I wasn't expecting a module release so soon, I'd better subscribe to that list... I am still trying to get a clear answer about the XORG_MACROS_VERSION(1.3). For example the xserver 1.7.99.2 claims it needs 1.3 but in fact it needs 1.4. I assume that's the reason why macros 1.4.1 was bumped and released.

I think xserver should be updated now that it has been released. There would be no harm done to also update all other modules, except perhaps raising the expectation that they will always be updated.

Thanks

On Tue, Dec 29, 2009 at 07:41:13AM -0800, Gaetan Nadon wrote:
> I wasn't expecting a module release so soon, I'd better subscribe to that
> list... I am still trying to get a clear answer about the
> XORG_MACROS_VERSION(1.3). For example the xserver 1.7.99.2 claims it needs 1.3
> but in fact it needs 1.4. I assume that's the reason why macros 1.4.1 was
> bumped and released.
> 
> I think xserver should be updated now that it has been released. There would be
> no harm done to also update all other modules, except perhaps raising the
> expectation that they will always be updated.

I think the best thing to do is to make sure macros are always released
before anything depends on them, and then deps also include a bump in
the required version in configure.ac.

(In reply to comment #30)
> I think the best thing to do is to make sure macros are always released
> before anything depends on them, and then deps also include a bump in
> the required version in configure.ac.

That's what we did with the XORG_DEFAULT_OPTIONS added in macros 1.3,
and it seemed to work fairly well - there's always some users caught
by surprise, but when configure tells them their xorg-macros is too
old, many people (unfortunately not all) figure out the fix on their own.