Item7453: Attachment metadata access too expensive; mtime not otherwise accessible

If TWiki::Func::getRevisionInfo is called for an attachment, the returned date is NOT the mtime (last modified time) of the attachment. Rather, it is the time of the last attach operation. (e.g. with RCS, it's the time of the last rcs checkin).

The returned date should be the modification time of the attachment as supplied by filedate when the attachment is saved. The attachment may have been modified much before it was attached to the topic.

It gets worse.

The attachment's mtime is stored in the attachment metadata in the topic. But getting it is highly inefficient.

To get the latest rev of an attachment, one calls readTopic to get the FILEATTACHMENT metada, then looks for the {date} attribute.

This means the whole topic is read and parsed, which is not good. But this is the best case.

Consider the case where one wants an older rev of the attachment. The FILEATTACHMENT data for the older attachment revision is in some older topic revision.

Topic and attachment revisions have no useful relationship to each-other; a topic will usually be saved many times between changes to an attachment.

Thus, the only way to get the correct metadata for a given attachment revision is to read every topic revision from the latest back toward topic rev 1, at each rev checking $meta->get( 'FILEATTACHMENT', $filename)->{version} to see if it matches the requested attachment version.

To see just how bad this is, consider a topic with 1,000 revisions, and an attachment with rev 1 attached at topic rev 2, and rev 2 attached at topic rev 10.

• Finding the mtime of the attachment at 'current' rev (rev 2) means reading the current version of the topic.
• Finding the mtime of the attachment's 'previous' rev (rev 1) means reading revisions 1000 - 2 -- 999 topic reads -- to get the metadata for the correct rev!
  One could do somewhat better with a binary search - except that doesn't work when topics have been reverted. (This can bring old attachment metadata forward in time, so the data isn't in chronological order - which a binary search would require.)

There are two fundamental issues:

* The attachment metadata should be stored somewhere else (e.g. working/metadata/Web/Topic/attachment), with a suitable API

* getRevisionInfo should be returning the mtime of the last revision. This is probably more tractable in the short term - get RCS to checkout with the correct mtime, and getRevisionInfo can then simply do a stat().

-- TWiki:Main/TimotheLitt - 2014-03-09

This is expected behavior, e.g. not a bug unless I miss something. The spec is as follows:

• Topic meta data:
  • The META:TOPICINFO meta data is a cache of the RCS file. This is done for performance so that the rcs file does not need to be accessed
  • If older revisions are requested it falls back to the RCS file.
• Attachment meta data:
  • The META:FILEATTACHMENT meta data has two purposes:
    • Index of attachments
    • Cache of attachment meta data - also done for performance
  • If older revisions are requested it falls back to the RCS file of the attachment, e.g. is reasonably fast.

TWiki records the upload time of the attachment, not the mtime. I think this is the expected behavior.

TWiki currently does not yet correlate the attachment revision with the topic revision. In a proper world, if you look at an older revision of a topic, liunks to attachments and embedded images should point to the attachment versions used at that time.

-- TWiki:Main.PeterThoeny - 2014-03-10

Also, stat() cannot be used because it would depend on the proper file timestamp. TWiki installations move around or merge (company merger) where timestamps can change, thus relying on file time stamp for TWiki data is not reliable.

-- TWiki:Main.PeterThoeny - 2014-03-10