Cairo spec for special characters in uploaded files and special file names:

• Filter out not supported chars (other than mixedAlphaNum or '.', '_', '-')
• Append .txt to dangerous file names such as .htaccess, *.pl etc

Dakar spec:

• Prevent upload with very technical message if {UploadFilter} or {NameFilter} regex found in filename

Ramifications of changing spec from permissive to restrictive behaviour:

• Upload of same filename no longer possible after upgrade, for example a person who uploads a weekly QA Meeting Minutes.doc file
• No longer possible to attach a file for customer support without specifically renaming the file, for example .htaccess
• Very technical message scares users:
  Your filename matched one of the following regular expressions, so it was rejected:
  {UploadFilter} /^(\.htaccess|.*\.(?i)(?:php[0-9s]?|phtm[l]?|pl|py|cgi))$/
  {NameFilter} /[\s\*?~^\$@%`"'&;|<>\x00-\x1f]/

TWiki is about easy collaboration. The permissive behaviour is to make it as easy as possible to work together. Getting content online quickly and easily is more important than a small filename change after upload.

To avoid confusion for users and to ease collaboration, Dakar should revert to the Cairo behaviour.

-- PTh -- See

• http://develop.twiki.org/~develop/cgi-bin/view/Bugs/Item822
• http://develop.twiki.org/~develop/cgi-bin/view/Bugs/Item443

When I think if FMEA - Faulure Mode Effect Analysis - mode I can see many problems with "merely" keeping "Cairo Compatibility", some of them leading to security issues.

• Uploading files whose names being with "." on UNIX/Linux system can have deleterious effects. Some of this may be direct, some may use this to give a wedge for further manipulation, such as symlinking. There are many documented attack modes based on this kind of vulnerability. q.v. the relevant on-line databases. Adding ".txt" is not enough.
  • Dakar does not guard against this either (only .htaccess) -- PTh
• File names with spaces lead to problems with interactions between browsers and servers.
  • Even under Cairo the name of the file gets munged. That can lead to problems for the unwary.
    • No, Cairo removed spaces on upload -- PTh

I absolutely agree about the error message. But then I think the presentation of the "oops" screens is sorely lacking in quite a number of ways.

-- AJA

What we call the file on disk and what we present it to the user as can be entirely different.

How about we HTML escape all (dubious) characters before writing them to disk?

-- MC

I also agree about the error message - it can be made a lot more friendly. But Anton hit the nail on the head. This change was intended to handle two criticisms of TWiki; that it changes file names on upload without telling the user, which is a fundamental no-no in interaction design, and that the .txt "solution" opens a potential security hole.

If I could have, I would have redesigned the entire interaction to give the user feedback, but in the end I didn't have the time or energy. The best form of interaction would be to filter the name in the upload screen, so that TWiki doesn't have to deal with bad filenames. But a bad filename is a failure mode. It is much better to let the user know that their filename isn't acceptable than to rename the file.

But the message needs rewriting, no question about that.

SVN 7451

CC

A future refinement might be to have the administrator write out an English wording for the regular expression. It'd need translating into locales though.

MC

Let me repeat: TWiki is about easy collaboration. The permissive behaviour is to make it as easy as possible to work together. Getting content online quickly and easily is more important than a small filename change after upload.

• Permissive behaviour on the part of software that permits easy compromise of corporate resourse is irresponsible behaviour.
  I've seen how people fled using TWiki after the security compromises related to the "grep". The possible compromises arrising from uploading arbitrary files and names are much more extensive and less likely to be aparrant to the administrator. -- AJA

The goal is to make it as easy ase possible to share content. The restrictive spec makes it harder.

On security, the Cairo permissive spec and the current restrictive spec does not guard against dot filenames, just .htaccess. This is a regex question, irrelevant to permissive vs restrictive question.

• Are you being deliberately obtuse, Peter? My point was more encompassing - that it should -- AJA
  • In this case I see no difference to filtering vs escaping -- PTh

This spec change requires existing users to rename files they used to be able to upload. Removing spaces from uploaded files is a lesser problem than not being able to upload a file.

We need to involve non-geeks into this decision process. We currently operate way too developer centric. Don't Make Me Think.

-- PTh

Strip the others out if you must but please don't strip spaces.

Spaces are:

1. very common
2. needed to visually grok content

I suggest we map them down to underscore (_) as that is not likely to cause problems.

Allowing spaces messes up files named like "RFP response from MegaCompany.doc"

-- MC

Supporting spaces is out of scope for Dakar, it involves too much code change. Replacing space with underscore on upload is a sensible workaround.

-- PTh

Hmmmm. Well, it was my tame non-geeks that got themselves into a terrible confusion because the files they uploaded had different names to the ones they started with. The change is something geeks can cope with, but non-geeks get really confused by. They would rather have their "wrong" filename rejected for good reason, so they don't make the same mistake again.

BTW the particular instance that triggered this change was a user who uploaded a file "Urgent Secret Report.doc". They mailed the big boss telling him they had attached the document as "Urgent Secret Report.doc" to a given topic - and big boss couldn't find it, because the filename had been changed. Big boss was made to look stupid in front of a client and went ballistic. If the rules for attachment naming were up front, clear and unambiguous, this wouldn't have happened.

CC

No further feedback in the last five days, so I'm assuming my rationale is understood and accepted, and am discarding this.

CC

This is a good example you bring, Crawford. I can bring other examples of new users who are intimidated after a failed upload, turning back to e-mail attachments with "this wiki thing is too complicated". Which case is more common? I bet by far the intimidated user.

Democracy works if the participants are representative for all stakeholders. In this is particilar case (and other cases here) we tend to favor the developer's point of view and forget about KISS, simply because users do not monitor the bugs web. We need to be get more user focused (which is what Jotspot and others are doing.)

I asked some users this question: "On file attach, the current release filters out problematic chars from the filename when you attach a file to twiki, such as %, @, (, ) etc. In other words,, upload succeeds without error, but the filename might change. What is more intuitive and useul, (1) refusing upload if there are problematic chars (asking user to rename first), or (2) make upload easily and live with different filename if there are problematic chars?" Replies:

• User J: "I'd think that you should upload without refusing but make it clear the name changed."
• User/developer S: "Upload and show message to user that filename changed. Ideally, upload and show a rename screen if illagal char in filename."
• User D: "Ask user to rename file."
• User M: "Upload without intervention, then show a rename screen. If there are problematic characters, I do not have to change the name of the local file I want to upload."

Don't Make Me Think. Ideally, attach should allow any filename, url-encode it, and reconstruct it on download. This is technically possible, but a too invasive change shortly before the release.

This is a low impact change avoiding introducing new bugs:

• Revert to the Cairo spec of permissive behaviour
• Change spaces to underscore

I am actioning on this in the next few days.

-- PTh

Of course, if you select your users carefully and phrase the question appropriately, you can get any response you want! I asked my niece if she thought all uploaded images should be changed into pictures of horses. She gave an unequivocal yes! ;-).

Seriously, though, this may be another case where an installer has to be able to choose behaviour; I really don't want a repeat of the scenario I described. Don't change my data! A boolean switch next to the {NameFilter} in TWiki.cfg would do the job - $TWiki::cfg{RenameIllegalUploads}, though we should seriously consider replacing this with an interactive strategy for Edinburgh.

BTW another use-case the arose yesterday. I had a script that was uploading pictures to my genealogical database, and had two images - LesDeGaris.gif and Les De Garis.gif. Fortunately Dakar blocked the upload of the second image. With Cairo behaviour it would have overwritten the first! (i.e. if you are going to rename uploads, make sure you change to a name that isn't in use. Not sure how you are going to handle a re-upload of the same file, though....)

CC

http://koala.ilog.fr/twikiirc/bin/irclogger_log/twiki?date=2005-11-25,Fri&sel=268#l264 discusses an approach for Ed and beyond: using HTTP encode to allow any filename.

Peters approach above for Dakar is reasonable and a good compromise.

In the distant future a notifications framework could be used to message the user (by email, messenger, by status topic) of things that have changed while they were not noticing.

MC

"reasonable and a good compromise" if and only if an upload of This File.gif doesn't overwrite a previous upload of This_File.gif.

CC

Total lack of action on this. Discarding.

CC

On my long to-do list. See also TWiki:Support.EmbeddedBlanksInAttachmentFilenames

-- PTh

This is now implemented, SVN 8239. Changed upload from restrictive to permissive behaviour, warning users with a friendly message in case filename has changed. See sample message. I did many tests and will do more tests tomorrow.

-- PTh