I have started testing the new tql based search. Ie. the type="query" search tested in practical
And I went for the simple use case first. I like the simple syntax. Much more logical than doing the same with the regex meta hack we have been used to.
The test I am doing is replacing some formatted searches where I only search for one value in one field.
But the new search has a severe performance problem. I have not timed it very accurately. Just counting the seconds. My meta regex searches take round 7 seconds. The tql search takes 12 seconds. Almost double up. That needs to get optimized. There is no good reason why such a simple search should be half the speed in the twl version as I assume it is in practical still implemented as a regex search.
Take a look for yourself.
I tested with the crond stopped. I repeated tests at least 10 times for each page. Back and forth. There is no doubt. The new search has a performance problem. Hopefully one that can be resolved.
If you benchmark on my server then remember that from XX:00 - XX:05 and XX:30 - XX:35, cron runs svn update, remove links, chown, pseudo-install and this makes the server very slow and impossible to use for timing tests. Also remember that the server runs on an ADSL line. When I test I have a gigabit network. That can mean a big difference. So you need to create a similar environment on your local machine to recreate.
--
TWiki:Main/KennethLavrsen - 01 Jun 2007
"I assume it is in practical still implemented as a regex search" - no. I spent all the time I had planned to spend on optimisation answering alternative syntax proposals, as it was clear that getting the syntax right was a greater issue than performance of a (potentially broken) implementation. At the moment the implementation is based on a linear, brute-force search through the topics. I did some experiments with using RE's to accelerate it, but decided that RE optimisation wasn't practical in the time I had available.
There are three possible approaches I am considering:
- "Winnowing" the set of topics that have to be brute-force searched by extracting and applying static RE's derived from the query
- A DBCache based search engine (raises the requirements on the install, but up to 10X faster than an RE search)
- A DBI store, which can be searched using SQL
--
CrawfordCurrie - 02 Jun 2007
Kenneth, please repeat your benchmarks, and close this report when you are happy with the performance.
CC
Looks very good. Measurements of the "Motion bug search" using ab 10 times.
- Patch04x01 regex search (more or less 4.1.2): 5.4 seconds
- MAIN regex search SVN 14112: 5.4 seconds.
- MAIN query search SVN 14112: 5.4 seconds.
Exactly the same. So excellent work. Closing.
KJL 13 Jun 2007