There are two issues with the holdovers in the scaled_font_map when font_faces
with destroy_funcs are involved:

1) It is inconvenient to the client that it cannot predict when its
   destroy_func on a font_face may be called (nor force the destruction of the
   font_face).

   This is particularly inconvenient for clients that are dynamic libraries
   (or modules) that want to be unloaded, as they have to ensure that the
   library stays in memory until the destroy_func is called, and then they
   must have a means to initiate an unload the library from the library itself
   (when the last destroy_func is called).

2) holdover scaled_fonts from font_faces with destroy_funcs that are
   unreferenced externally are unlikely to be used again, so they are often
   being heldover unnecessarily.

I can understand why scaled_font holdovers are necessary:

  References are not in general held to scaled_fonts, but instead scaled_fonts
  (and their glyph caches) are found when needed from font_face, font_matrix,
  ctm, context font_options, and surface font_options.

  Even if the client were to try to hold onto scaled_fonts that it expects to
  be reused, getting this exactly right would be a little complex due to the
  interaction of the surface font_options.

But the situation is different with font_faces:

  font_faces that are likely to be reused are much more likely to have
  external references.

  It is also easy for the client to keep references to font_faces and thus
  have some control over the memory use / speed balance associated with
  caching fonts.

holdover scaled_fonts are only useful when they are likely to be used again,
but the number of external references to font_faces provides the best
indicator of whether they will be used again.

  Perhaps some apps may release all references to font_faces created from
  filename/id (and all references to their scaled_fonts) but would still be
  likely to want the same font again soon.  But, for font_faces created from
  FT_Face (and other faces with destroy_funcs) at least, the lack of a
  reference is a good indicator that the same font_face will not be needed
  again soon.

So I'm imagining a situation where each holdover scaled_font no longer owns
its font_face but is recorded on the font_face (or unscaled_font).

  When the scaled_font is referenced again it would remove itself from the
  font_face's holdover list and take a reference to the font_face.

  If the last reference to the font_face is removed then it would arrange
  destruction of all the scaled_fonts in its holdover list.

  I imagine a global holdover list would still be required to keep the current
  behavior of maintaining a limit on the total number of holdover scaled_fonts
  by destroying the least recently used.

(The motivation for this began here:
 https://bugzilla.mozilla.org/show_bug.cgi?id=458169#c15 )