Item5135: EditTablePlugin must disable initsort when editing table TABLE tag to work with new move row feature

When you have a table with initsort in the TABLE tag and then hit the edit button the resulting table in edit mode has all the row shuffled.

I cannot see any system in the sequence. It make editing a table almost impossible if the table contains ordered data.

This "kills" our status report application where we track maybe 50-100 milestones so I mark this one urgent because I am sure many will be affected by this bug.

Either the edit table needs to be presented in the same sequence as the raw table, or sorted correctly as given by initsort. In 4.1.2 the sequence was the raw table content and that is fine.

EDITTABLE above TABLE

Milestone	Plan	Forecast	Actual
Some Start Milestone (1)	30 Jun 2007		14 Sep 2007
Doc Input (2)	12 Sep 2007		12 Sep 2007
Doc Plan (3)	26 Sep 2007		26 Sep 2007
Q Plan (4)	10 Oct 2007		30 Nov 2007
Management Decision (5)	10 Oct 2007		10 Oct 2007
Management Review (6)	12 Oct 2007		12 Oct 2007
Senior Management Babble (7)	17 Oct 2007	13 Dec 2007
SW Test 1 (8)	12 Dec 2007	12 Dec 2007
SW Test 2 (9)	11 Jan 2008	11 Jan 2008
Internal test (10)	11 Jan 2008	11 Jan 2008
Upgrade stuff (11)	11 Jan 2008	11 Jan 2008
Test more stuff (12)	01 Feb 2008	01 Feb 2008
Stuff tested and integrated (13)	21 Mar 2008	21 Mar 2008
Upgrade next version (14)	21 Mar 2008	21 Mar 2008
Test next version (15)	11 Apr 2008	11 Apr 2008
Next version tested (17)	11 Apr 2008	11 Apr 2008
Write more doccu (16)	05 May 2008	05 May 2008
SW Final version (18)	16 May 2008	16 May 2008
SW Final final version (19)	30 May 2008	30 May 2008
Upload release (21)	20 Jun 2008	20 Jun 2008
Build release (20)	21 Jun 2008	09 Jun 2008

TABLE above EDITTABLE

Milestone	Plan	Forecast	Actual
Some Start Milestone (1)	30 Jun 2007		14 Sep 2007
Doc Input (2)	12 Sep 2007		12 Sep 2007
Doc Plan (3)	26 Sep 2007		26 Sep 2007
Q Plan (4)	10 Oct 2007		30 Nov 2007
Management Decision (5)	10 Oct 2007		10 Oct 2007
Management Review (6)	12 Oct 2007		12 Oct 2007
Senior Management Babble (7)	17 Oct 2007	13 Dec 2007
SW Test 1 (8)	12 Dec 2007	12 Dec 2007
SW Test 2 (9)	11 Jan 2008	11 Jan 2008
Internal test (10)	11 Jan 2008	11 Jan 2008
Upgrade stuff (11)	11 Jan 2008	11 Jan 2008
Test more stuff (12)	01 Feb 2008	01 Feb 2008
Stuff tested and integrated (13)	21 Mar 2008	21 Mar 2008
Upgrade next version (14)	21 Mar 2008	21 Mar 2008
Test next version (15)	11 Apr 2008	11 Apr 2008
Next version tested (17)	11 Apr 2008	11 Apr 2008
Write more doccu (16)	05 May 2008	05 May 2008
SW Final version (18)	16 May 2008	16 May 2008
SW Final final version (19)	30 May 2008	30 May 2008
Build release (20)	09 Jun 2008	09 Jun 2008
Upload release (21)	20 Jun 2008	20 Jun 2008

TABLE before EDITTABLE

Milestone	Plan	Forecast	Actual
Some Start Milestone (1)	30 Jun 2007		14 Sep 2007
Doc Input (2)	12 Sep 2007		12 Sep 2007
Doc Plan (3)	26 Sep 2007		26 Sep 2007
Q Plan (4)	10 Oct 2007		30 Nov 2007
Management Decision (5)	10 Oct 2007		10 Oct 2007
Management Review (6)	12 Oct 2007		12 Oct 2007
Senior Management Babble (7)	17 Oct 2007	13 Dec 2007
SW Test 1 (8)	12 Dec 2007	12 Dec 2007
SW Test 2 (9)	11 Jan 2008	11 Jan 2008
Internal test (10)	11 Jan 2008	11 Jan 2008
Upgrade stuff (11)	11 Jan 2008	11 Jan 2008
Test more stuff (12)	01 Feb 2008	01 Feb 2008
Stuff tested and integrated (13)	21 Mar 2008	21 Mar 2008
Upgrade next version (14)	21 Mar 2008	21 Mar 2008
Test next version (15)	11 Apr 2008	11 Apr 2008
Next version tested (17)	11 Apr 2008	11 Apr 2008
Write more doccu (16)	05 May 2008	05 May 2008
SW Final version (18)	16 May 2008	16 May 2008
SW Final final version (19)	30 May 2008	30 May 2008
Build release (20)	09 Jun 2008	09 Jun 2008
Upload release (21)	20 Jun 2008	20 Jun 2008

TABLE after EDITTABLE

Milestone	Plan	Forecast	Actual
Some Start Milestone (1)	30 Jun 2007		14 Sep 2007
Doc Input (2)	12 Sep 2007		12 Sep 2007
Doc Plan (3)	26 Sep 2007		26 Sep 2007
Q Plan (4)	10 Oct 2007		30 Nov 2007
Management Decision (5)	10 Oct 2007		10 Oct 2007
Management Review (6)	12 Oct 2007		12 Oct 2007
Senior Management Babble (7)	17 Oct 2007	13 Dec 2007
SW Test 1 (8)	12 Dec 2007	12 Dec 2007
SW Test 2 (9)	11 Jan 2008	11 Jan 2008
Internal test (10)	11 Jan 2008	11 Jan 2008
Upgrade stuff (11)	11 Jan 2008	11 Jan 2008
Test more stuff (12)	01 Feb 2008	01 Feb 2008
Stuff tested and integrated (13)	21 Mar 2008	21 Mar 2008
Upgrade next version (14)	21 Mar 2008	21 Mar 2008
Test next version (15)	11 Apr 2008	11 Apr 2008
Next version tested (17)	11 Apr 2008	11 Apr 2008
Write more doccu (16)	05 May 2008	05 May 2008
SW Final version (18)	16 May 2008	16 May 2008
SW Final final version (19)	30 May 2008	30 May 2008
Build release (20)	09 Jun 2008	09 Jun 2008
Upload release (21)	20 Jun 2008	20 Jun 2008

-- TWiki:Main/KennethLavrsen - 14 Dec 2007

I prefer to remove sorting from the table in edit mode.

-- TWiki:Main.ArthurClemens - 15 Dec 2007

There is system in it

I added some numbers to the text. Try and edit. It seems the table gets sorted so that each line is first given a number.

When you edit some label for each row indicates the sequence. This label is then sorted. But sorted alphanetically. So 10 comes before 9.

It cannot be hard to fix. Either the label must be sorted ignorring the prefix or the number part of the label must be padded by some zeroes.

-- TWiki:Main/KennethLavrsen - 15 Dec 2007

Removing the sort while editing is probably the best yes

-- TWiki:Main.KennethLavrsen - 15 Dec 2007

An extra info. In 4.1.2 initsort actually works also in edit mode. So maybe we should keep that. Sorting by clicking the header does not work in 4.1.2. There is an old bug item on that. You loose what you edit by hitting the header while in edit mode. So that sort mechanism should just be removed. But initsort working, that is kind of cool - so if it is possible to fix then...

-- TWiki:Main.KennethLavrsen - 15 Dec 2007

Thinking about it. The new edittable feature of moving rows means that we should turn off the initsorting while editing also. Otherwise those two features goof up.

-- TWiki:Main.KennethLavrsen - 15 Dec 2007

I have been playing a little and I think I have a fix which modifies the TablePlugin.

My idea is to disable the sorting for a table when the cgi parameter etedit in the EditTablePlugin is defined AND the ettablenr matches the current table number.

A hack but I see no other way without disabling all sorting on all tables in same topic while editing.

I need to test many hours. I need to test without EditTablePlugin enabled and many different combinations of topics.

-- TWiki:Main.KennethLavrsen - 15 Dec 2007

To insert disableallsort in the TABLE parameters, add to line 288 of ETP Core.pm:

# when editing make sure the table is not sorted            
$_ =~ s/(}%)/ disableallsort="on"$1/ if $doEdit;

-- TWiki:Main.ArthurClemens - 16 Dec 2007

Then, at line 701, add:

$text .= hiddenField( $preSp, 'sort', 'off', "\n" );

-- TWiki:Main.ArthurClemens - 16 Dec 2007

In TablePlugin I am adding in Core.pm line 148

    # If EditTablePlugin is installed and we are editing a table, the CGI parameter 'sort' is
    # defined as "off" to disable all header sorting and the 'disableallsort' TABLE parameter
    # is added to disable initsort in the table that is being edited. 
    my $query = TWiki::Func::getCgiQuery();
    if ( $query->param( 'sort' ) eq 'off' ) {
        undef $sortAllTables;
    }

    $tmp = $params{disableallsort};
    if ( defined $tmp && $tmp ne '' ) {
        undef $sortAllTables;
        undef $initSort;        
    }

Thanks for the help Arthur. Much better solution than what I proposed. And my original proposal would not work anyway.

I will test it carefully Sunday and write the needed doccu also in the code before I check it in.

-- TWiki:Main.KennethLavrsen - 16 Dec 2007

To use the url parameter (hidden form field) sort="off" in TablePlugin, add below this code:

    # sorting may be disabled by url parameter sort=off
    my $cgi = TWiki::Func::getCgiQuery();
    my $cgiSort = $cgi->param('sort');
    if ( defined $cgiSort && $cgiSort eq 'off' ) {
      undef $sortAllTables;
   }

-- TWiki:Main.ArthurClemens - 16 Dec 2007

This way of hacking the TABLE tag will not work.

The disableallsort="on" gets SAVED with the table. Each time you save another gets appended.

-- TWiki:Main.KennethLavrsen - 16 Dec 2007

Ah. That was also easy to fix. Just changes the condition from only looking at $doEdit to

$_ =~ s/(}%)/ disableallsort="on"$1/ if ( $doEdit && !$doSave );

I will continue testing. It pays off to test for many hours with many combinations and many repetitions.

-- TWiki:Main.KennethLavrsen - 16 Dec 2007

All tested, unit tests updated so they pass.

Uploaded to t.o.

Changing title of bug so it can go into release note because the disabling of sort is a spec change from 4.1.2. But needed because the new nice feature of moving rows up and down requires a predictable sequence.

-- TWiki:Main.KennethLavrsen - 17 Dec 2007

We missed an important detail.

My test case that showed the problem was fixed. But when you swap TABLE and EDITTABLE the fix Arthur and I made together fails because at the time where we decide to add the disableallsort="on" to the TABLE tag we need to have passed the EDITTABLE tag for variables such as $doedit are defined.

I have tried some different models like appending based on $query->param('ettablenr') but also this parameter gets set when we pass the EDITTABLE. I guess the fix will include passing by the TABLE and if the next line is an EDITTABLE then go back and append the disableallsort="on". But doing this requires that we know 100% what is happening.

There are 4 testcases to consider. I have added them at the top of this bug item. They are TABLE before EDITTABLE and EDITTABLE before TABLE. And the two either on two lines or same line.

And on the same line an extra danger is: which }% is the right one to find with the regex? What is there is a twiki variable inside the TABLE tag or EDITTABLE tag?

The parsing going on in the plugin is getting pretty "long haired".

-- TWiki:Main.KennethLavrsen - 09 Jan 2008

There is something very goofy in the code.

If TABLE is above EDITTABLE the condition that adds the disableallsort="on" is never met.

If EDITTABLE is on the same line as TABLE and EDITTABLE comes before TABLE the disableallsort="on" is added to the EDITTABLE tag. This is easy to fix. But even when fixed it seems the $_ is never used as part of the returned result. And this also hits the last condition when TABLE comes before EDITTABLE on the same line.

Something is very goofy with this code and I am getting concerned about other hidden bugs in that horrible processText function.

I do not feel comfortable release TWiki without this being fixed and reviewed by more than one person. And I am travelling the days up to the release so I cannot put more hours in this.

-- TWiki:Main.KennethLavrsen - 10 Jan 2008

I had hoped this had been resolved for 4.2.0.

The problem with this is that unless that EDITTABLE is before TABLE and on two seperate lines, and you use initsort, the plugin totally goofs up the sorting because it counts the columns including the first move icon column. So when you hit edit you get what appears to be a randomly sorted table. I tried to fix this before I went to London but the code had become too complicated for a quick fix. Need help.

-- TWiki:Main.KennethLavrsen - 14 Jan 2008

Fixed, finally. In order to add the parameter disableallsort="on" to the TABLE tag I have converted the list of lines to an array, which makes it easier to change the value of a preceding line.

-- TWiki:Main.ArthurClemens - 15 Mar 2008