|
Thx for our suggestions, they will be invaluable in my efforts to code more optimally.
In my own search, I have managed to find a 4x speed improvement by re-coding how tasks are inserted into the tree control. I was still using an old algorithm which constructed the data structure and tree in one hit. Once I separated these processes I was able to work much more efficiently. So now a 2000 task file loads in 6 seconds. I would be very interested to see what times you get if you run your script on a similarly sized tasklist.
|
|
|
|
|
Specified file
File read time: 0 (it might have been in my file cache though - AHK timing is accurate to about 10ms)
XML Load time: 125
Object Create time: 656
Gui Creation time: 156
Total load time: 937
Total Save time: 78
UniCode save: 282 (much slower than ANSI)
Specified file - converted to unicode
File read time: 31
XML Load time: 94
Object Create time: 656
Gui Creation time: 141
Total load time: 922
Total Save time: 93
UniCode save: 328
Note that all timings are with the Unicode build of AHK
The code
#SingleInstance force
File=C:\2000_tasks.tdl
StartTime:=A_TickCount
StartTime2:=A_TickCount
FileRead, xml, %File%
result:="File read time:`t" . (A_TickCount -StartTime) . "`n"
StartTime:=A_TickCount
TDL:=loadXML(XML)
result.="XML Load time:`t" . (A_TickCount -StartTime) . "`n"
test:=TDL.childNodes.Item(0)
for attrib In test.attributes
if attrib.name="encoding"
{
attrib.nodevalue:="UTF-16" ;uncomment to use unicode in saving - currently I have no other workaround
}
StartTime:=A_TickCount
TaskList:=XML2Object(TDL.childnodes)
result.="Object Create time:`t" . (A_TickCount -StartTime) . "`n"
StartTime:=A_TickCount
Gui, Add, TreeView,gTreeviewHandler vTreeViewTasks w400 h500
AddTree(TaskList)
Gui, Show ; Show the window and its TreeView.
result.="Gui Creation time:`t" . (A_TickCount -StartTime) . "`n"
result.="Total load time:`t" . (A_TickCount -StartTime2) . "`n"
;StartTime:=A_TickCount
;AddTasks(TDL,100)
;result.="Add 100 tasks:`t" . (A_TickCount -StartTime) . "`n"
test:=TDL.childNodes.Item(0)
for attrib In test.attributes
if attrib.name="encoding"
{
attrib.nodevalue:="windows-1252" ;uncomment to use unicode in saving - currently I have no other workaround
}
StartTime:=A_TickCount
TDL.save("C:\Hercules\My Documents\Todolist\QA Ansi (Saved).tdl")
result.="Total Save time:`t" . (A_TickCount -StartTime) . "`n"
test:=TDL.childNodes.Item(0)
for attrib In test.attributes
if attrib.name="encoding"
{
attrib.nodevalue:="UTF-16" ;uncomment to use unicode in saving - currently I have no other workaround
}
StartTime:=A_TickCount
TDL.save("C:\Hercules\My Documents\Todolist\QA Uni (Saved).tdl")
result.="UniCode save:`t" . (A_TickCount -StartTime) . "`n"
ClipBoard:=Result
msgbox,%result%
exitapp
TreeviewHandler:
if (A_GuiControl="TreeViewTasks") and (A_GuiEvent="DoubleClick")
{
TaskID:=FindInTree(TaskLIst,A_EventInfo)
Gui, Destroy
tmp:="""" File """"
}
return
FindInTree(Tasklist,TVID)
{
enum:=TaskList._NewEnum()
while enum[ID, Task]
if isObject(Task)
{
if (Task.TVID=TVID)
return Task.ID
enum2:=Task._NewEnum()
while enum2[cID, Child]
if isObject(child)
{
GetID:=FindInTree(Task,TVID)
if (GetID<>"")
return GetID
}
}
return ""
}
XML2Object(nodes,level=0,parent="")
{
NewTasks:=Object()
for node in nodes
{
NodeName:=node.nodename
NodeO:=Object()
for attrib In node.attributes
NodeO[attrib.name]:=attrib.nodevalue
if (NodeName="Task")
NodeName:=NodeO.ID
if node.hasChildNodes
{
ChildNodes:=XML2Object(node.childNodes,level+1,nodename)
{
enum:=ChildNodes._NewEnum()
while enum[ID, Child]
{
NodeO[ID]:=Child
}
}
}
NewTasks[NodeName]:=NodeO
}
return NewTasks
}
loadXML(ByRef data)
{
o := ComObjCreate("MSXML2.DOMDocument.6.0")
o.async := false
o.loadXML(data)
return o
}
AddTasks(TaskList,numtasks=1)
{
Loop,%numtasks%
{
Random, TaskTitle , 0, 2147483647
Random, DueDate, 40000, 41000
TDL_Add_TopLevel(TaskList,TaskTitle,DueDate)
}
}
TDL_Add_TopLevel(TaskList,Title,DueDate="")
{
NextID:=TaskList.childNodes.Item(1).getAttribute("NEXTUNIQUEID")
MyNode:=TaskList.createNode(1,"Task","")
MyNode.setattribute("Title",Title)
Mynode.setattribute("ID",NextID)
MyNode.setattribute("DueDate",DueDate)
MyNode.setattribute("Magic","Added by TDL Benchmark")
TaskList.childNodes.Item(1).appendChild(myNode)
TaskList.childNodes.Item(1).setAttribute("NEXTUNIQUEID",NextId+1)
}
AddTree(Tasklist,Parent="")
{
enum:=TaskList._NewEnum()
while enum[ID, Task]
if isObject(Task)
{
if (Parent="")
{
if (ID ="TODOLIST")
NewP:=TV_Add(Task.Filename)
}
else
if task.title<>""
NewP:=TV_Add(Task.Title,Parent)
Task.TVID:=NewP
enum2:=Task._NewEnum()
HasChild=0
while enum2[cID, Child]
if isObject(child)
HasChild=1
if HasChild=1
AddTree(Task,NewP)
}
}
|
|
|
|
|
Interesting that you also experienced a substantial slow-down using Unicode. Anyway I hope to get to the bottom of it tonight.
|
|
|
|
|
I thought so too. At first I thought that it was due to the increased file size, since the MSXML object at that stage is fully prepared, and only needs to be dumped to a file (even the Unicode conversion is supposed to be done). The slowdown is much greater than x2 even, which is unexpected.
It is impossible to debug though since this happens in one call to MSXML - which is a black box for me.
|
|
|
|
|
|
Thanks Chris,
My conclusions:
Most optimisations seems to propose:
1)More CPUs
2)More threads
3)Smarter node selection
1 is not a solution for TDL, 2 possibly since multi-core processors is becoming more abundant, however the increased complexity of a multi-threaded solution is not a recommendation that I will make.
Smarter node selection for TDL is irrelevant as the whole tasklist needs to be loaded (it may be an option for AHK-based helper apps which may want to change only one task at a time - but this is not a consideration at the moment)
then http://www.hanselman.com/blog/AnOldieButAGoodieMSXMLAndFreeThreadedDomDocumentVsDomDocument.aspx[^] proposes
var doc = new ActiveXObject("MSXML2.DOMDocument");
doc.validateOnParse = false;
doc.resolveExternals = false;
doc.preserveWhiteSpace = false;
doc.load("somexml.xml");
which turns of some validation (I do not know what). I tested this and it led to an about 10% improvement - which is insufficient and does not address the nonlinearity problem of loading/saving.
http://www.xml.com/pub/a/2002/06/05/msxml4.html[^] proposes to use
objXML.setProperty("NewParser", true);
which apparently also turns of some validation, but my tests does not show any noticeable improvement.
This
http://msdn.microsoft.com/en-us/library/ms950760.aspx[^]
makes a few suggestions covered above, but also the statement:
"The fastest way to walk the tree is to avoid the children collection and any kind of array access. Instead, use firstChild and nextSibling:"
I currently use the children collection - so I will try and see what the speed improvement is (I read this article on my phone while travelling - but somehow I missed this part!)
It also highlights the slowdown in TDL with attribute heavy XML objects - which makes me wonder if an optimisation could be to not save defaults/blank strings (although this may cause a lot of overhead in the process as well - and it may not be easy to achieve)
Lastly: http://msdn.microsoft.com/en-us/magazine/cc163436.aspx[^]
I cannot test this, but this looks promising. Flexibility always adds overhead and a lot of the options in MSXML is not required by TDL
|
|
|
|
|
Hi,
regarding:
>>Smarter node selection for TDL is irrelevant as the whole tasklist needs to be loaded (it may be an option for AHK-based helper apps which may want to change only one task at a time - but this is not a consideration at the moment)<<
Would it make sense to use the sax parser instead of the DOM-Parser (msxml) as sax is working in serial manner?
http://msdn.microsoft.com/en-us/library/ms763715[^]
Regards
Chris
modified on Thursday, July 28, 2011 10:49 AM
|
|
|
|
|
I don't know if it might boost loading speeding - but SAX is for loading and not saving.
This will mean that you have a SAX loader and a DOM saver.
Biggest problem is in saving in any case, so I do not think switching the loading parser to SAX will add a lot of value.
|
|
|
|
|
capital H wrote: It also highlights the slowdown in TDL with attribute heavy XML objects - which makes me wonder if an optimisation could be to not save defaults/blank strings I like this idea. All it would mean would be to pass in a default value when building the DOM that when compared against would decide whether the attribute needed to be added. Even with text values, the most likely default value would be an empty string which is easy to test for efficiently.
|
|
|
|
|
I am stunned
From:
http://msdn.microsoft.com/en-us/library/ms950760.aspx[^]
"The number of Unicode characters in text content (including attribute values). Note that loading single-byte ANSI text into memory results in twice the number, because all text is stored as Unicode characters, which are two bytes each."
Now assuming this is true (written by "Chris Lovett is a program manager for Microsoft's XML team." - no reason to believe that it is not) I simply cannot understand the deterioration in unicode performance in both our implementations.
|
|
|
|
|
Something else that you can consider (although this will not help for speed) is to save the number of tasks in a tasklist in the TODOLIST line, and then to display some sort of progress indicator. At least then a user will know that todolist.exe is still working, and will know how long to wait (stretch you back/go and make coffee/go and have lunch)
|
|
|
|
|
|
Get it Here.
Changes
- 'Email Tasks' task selection improved
- Proper tdl file importer/exporter added with option to reset the task creation date
- Calendar plugin has new preferences dialog with improved display options
- Task selection history bug fixed
- Carriage return added between tasks exported/copied 'As Text'
- 'Your Language.csv' file added to 'Translations' sub-folder - ready for translation
Remember: This is still an Alpha build so backup your tasklists.
|
|
|
|
|
Hi Dan,
.dan.g. wrote: 'Email Tasks'
You can find this in previous versions too:
* Alt+F -> 'Email tasks ...'
* Right mouse click on a tab -> 'Send To ...'
* Right mouse click on a task in a tasklist -> an option to email or send a task is not offered.
(This is the context menu where I'd expect to find an "email task' option).
I'd like to suggest "email list / task' for all three, since it is possible to send a whole list too and not only selected tasks or a single task.
Cheers,
Jochen
|
|
|
|
|
Why don't I just route all the 'Send To...' commands (including right-clicking on selected Tasks) to the same selection dialog?
ie. When right-clicking on selected tasks it will initialize the 'Selection' radio button, else it selects 'All Tasks'.
|
|
|
|
|
.dan.g. wrote: Why don't I just route all the 'Send To...' commands (including right-clicking on selected Tasks) to the same selection dialog?
I'm not sure if I get you wright, but these two options
* Alt+F -> 'Email tasks ...'
* Right mouse click on a tab -> 'Send To ...'
are leading already to the same selection dialogue, aren't they?
Theoretically there are a lot of options like:
* right clicking on a single task -> leads to a context menu -> send task(s)
* selecting two and more tasks (but all the tasks in the list) -> same menu -> send task(s)
* selecting all the tasks -> send tasklist.
I think the context menu the user gets at the moment by clicking on a tab or using Alt+F is better, because it always offers all the options (send task, send list).
The only option that is missing is the 'send to...' when you right-click on a task.
And that should - as you mentioned - lead to the same context menu. Sure.
Last but not least:
I prefer "email task / list" instead of "send to" because it's more obvious than "send to".
"Send to" could have the meaning of "send to another list" or "send to printer", too. With "email tasks" it's WYSIWIG.
"Send to" would only be the right choice if you are planning to offer "send to another list" or "sending to printer". But still: I assume it would be better to have three items in the context menu instead of one item "send to" that leads to sub-menus.
|
|
|
|
|
I have been out of circulation for a while but finally completed my ANSI vs UNICODE AHK benchmarker.
Conclusion:
1. Opening TDL, opening a tasklist and most of the sorting seems to be at the same speed (A vs U)
2. Sorting by due date seems to be much slower for Unicode, but is still pretty quick.
3. There does not seem to be a difference for TDLU in opening a unicode file or an ANSI file (the first opening time benchmark is an ANSI file)
4. Saving tasklists is much slower for Unicode than for ANSI
The code:
UPDATED:
Longer delays between sending keystrokes
Sorting combined and looped to create more accurate timing
Nicer output
;http:
xml=
(
<?xml version="1.0" encoding="windows-1252"?>
<TODOLIST FILENAME="" PROJECTNAME="" FILEFORMAT="9" NEXTUNIQUEID="1" LASTMODIFIED="2011-07-21" FILEVERSION="640" EARLIESTDUEDATE="40745.00000000">
</TODOLIST>
)
while PathToTDLA=""
FileSelectFile,PathToTDLA,1,,Please select the path to the ANSI TODOLIST.exe,*.exe
while PathToTDLU=""
FileSelectFile,PathToTDLU,1,,Please select the path to the Unicode TODOLIST.exe, *.exe
if WinExist("AbstractSpoon")
{
MsgBox, Please close all ToDoList windows before running the benchmark
exitapp
}
setTitlematchMode,2
run, %PathToTDLA%
winwait, AbstractSpoon
SendMessage, 0x111, 32853,,,AbstractSpoon ; close all tasklists = which creates a new tasklist as well
sleep,500
winclose, AbstractSpoon
WinWaitClose
run, %PathToTDLU%
winwait, AbstractSpoon
SendMessage, 0x111, 32853,,,AbstractSpoon ; close all tasklists = which creates a new tasklist as well
sleep,500
winclose, AbstractSpoon
WinWaitClose
NumTasks:=4500
StartTime:=A_TickCount
loop,1
{
Result1:=benchmark(PathToTDLA,NumTasks)
Result2:=benchmark(PathToTDLU,NumTasks)
result.= "Benchmark " . NumTasks . " tasks `tANSI `t`t`tUnicode `n" . Combine(result1,Result2)
;Result.=benchmark(PathToTDLU,NumTasks,"UniCode")
NumTasks*=2
}
Result.="Total time to run benchmarks:`t" . (A_TickCount- StartTime)/1000 . "`n"
ClipBoard:=Result
Msgbox, Results in clipboard. Please paste in TDL forum
BenchMark(TDLExe,Num)
{
Random, , 1234 ;consistent random number generation - now everyone uses the same tasklists
StartTime := A_TickCount
global xml
TDL:=loadXML(XML)
AddTasks(TDL,Num)
TDL.save(A_Temp . "\BenchMark.tdl")
Result=
;Result.= "Creating tasklist (Debug info - not official benchmark): `t" . (A_TickCount- StartTime)/1000 . "`n"
StartTime := A_TickCount
run, %TDLExe%
winwait, AbstractSpoon
Result.= "Opening TDL: `t`t`t" . (A_TickCount- StartTime)/1000 . "`n"
SendMessage, 0x111, 33238 ,,,AbstractSpoon ;Task Tree View
StartTime := A_TickCount
run, %TDLExe% %A_Temp%\BenchMark.tdl
winwait, BenchMark
Result.= "Opening TaskList: `t`t" . (A_TickCount- StartTime)/1000 . "`n"
SendMessage, 0x111, 32798,,,AbstractSpoon ;Create new task
sleep,500
send {Enter}
sleep,500
send {del} ; and delete
sleep,500
StartTime := A_TickCount
SendMessage, 0x111, 32816,,,AbstractSpoon ;Save
winwait, BenchMark%A_Space%-
Result.= "Saving TaskList: `t`t" . (A_TickCount- StartTime)/1000 . "`n"
SendMessage, 0x111, 33151 ,,,AbstractSpoon ;unsorted
StartTime := A_TickCount
loop,10
{ SendMessage, 0x111, 33131 ,,,AbstractSpoon ;Sort by title
SendMessage, 0x111, 33151 ,,,AbstractSpoon ;unsorted
SendMessage, 0x111, 33128 ,,,AbstractSpoon ;Due date
}
Result.= "Sorting (10 loops): `t" . (A_TickCount- StartTime)/1000 . "`n"
StartTime := A_TickCount
SendMessage, 0x111, 32853,,,AbstractSpoon ; close all tasklists = which creates a new tasklist as well
winwait (untitled
StartTime := A_TickCount
SendMessage, 0x111, 57616,,,AbstractSpoon ; opens most recent open tasklist
winwait, BenchMark
Result.= "Re-opening TaskList: `t" . (A_TickCount- StartTime)/1000 . "`n"
SendMessage, 0x111, 32798,,,AbstractSpoon ;Create new task
sleep,500
send {Enter}
sleep,500
send {del} ; and delete
sleep,500
StartTime := A_TickCount
SendMessage, 0x111, 32816,,,AbstractSpoon ;Save
winwait, BenchMark%A_Space%-
Result.= "Re-saving TaskList: `t" . (A_TickCount- StartTime)/1000 . "`n"
SendMessage, 0x111, 32853,,,AbstractSpoon ; close all tasklists = which creates a new tasklist as well
sleep,500
winclose, AbstractSpoon
WinWaitClose
result.="`n"
return result
}
return
AddTasks(TaskList,numtasks=1)
{
Loop,%numtasks%
{
Random, TaskTitle , 0, 2147483647
Random, DueDate, 40000, 41000
TDL_Add_TopLevel(TaskList,TaskTitle,DueDate)
}
}
TDL_Add_TopLevel(TaskList,Title,DueDate="")
{
NextID:=TaskList.childNodes.Item(1).getAttribute("NEXTUNIQUEID")
MyNode:=TaskList.createNode(1,"Task","")
MyNode.setattribute("Title",Title)
Mynode.setattribute("ID",NextID)
MyNode.setattribute("DueDate",DueDate)
MyNode.setattribute("Magic","Added by TDL Benchmark")
TaskList.childNodes.Item(1).appendChild(myNode)
TaskList.childNodes.Item(1).setAttribute("NEXTUNIQUEID",NextId+1)
}
loadXML(ByRef data)
{
o := ComObjCreate("MSXML2.DOMDocument.6.0")
o.async := false
o.loadXML(data)
return o
}
Combine(String1, String2)
{
result=
Loop,parse, String2,`n
S%A_Index%:=substr(A_LoopField,instr(A_LoopField,"`t",false,-0))
Loop,parse, String1,`n
result.=A_LoopField . "`t" . S%A_Index% . "`n"
return result
}
modified on Friday, July 22, 2011 3:01 AM
|
|
|
|
|
Awesome, thanks _so_ much.
Hmm... Script causes 'Delete' confirmation dialog to pop up as well as 'Save' dialog, and when I respond to them it sort gets stuck and I have to kill ToDoList.exe.
Any thoughts?
ps. Despite this, thx for the benchmark results comparisons.
pps. Can you make the ini files available for download?
.dan.g.
AbstractSpoon Software
abstractspoon2_at_optusnet_dot_com_dot_au
modified on Thursday, July 21, 2011 8:35 PM
|
|
|
|
|
My script will get stuck as there is several places where it waits for the title to change etc. However I cannot understand why it will have the dialog popups for Delete and Save as I use the message IDs from Resource.h to communicate with ToDoList.exe. As long as the IDs between the build are the same, no "delete" or "Save" dialog should ever be displayed since I use:
Close all tasklists
Save Tasklist (not save as)
Open tasklist is done via command line
Create new task
Sort by title
sort by duedate
revert to unsorted
Open most recent tasklist
The only keyboard commands that I send is when I create a new task: I send an "Enter" and "Delete" (essentially I do this to force a changed file, I need a * in the title) - which can possibly be send to the wrong control, but I doubt it - I will change the code in any case to send to the treeview/listview.
ps. It is a pleasure.
pps. I do not follow, which INI files? My script does not make use of any INI files.
|
|
|
|
|
My own simple test for saving was:
1. 4500 tasks as Ansi in 6.2.7 (Ansi build) = 5 seconds
2. 4500 tasks as Ansi in 6.3.a8 (Unicode build) = 15 seconds
3. 4500 tasks as Unicode in 6.3.a8 (Unicode build) = 25+ seconds
I suspect that what's happening is that there's a lot of Ansi->Unicode->Ansi string conversions going on.
This will always cause a slowdown when 'Unicode to Ansi' saving(2), but I should be able to get the 'Unicode to Unicode' saving(3) to be as fast as the 'Ansi to Ansi' saving(1) notwithstanding a slight increase due to the increased Unicode file sizes.
|
|
|
|
|
I have heard that because most modern OSs is built on unicode and that unicode operations are hence quicker (obviously disk operations will be slower though).
PS: how do you choose in the UNICODE build what filetype to save as? My script just used a plain vanilla save, but I cannot find an option anywhere in ToDoListU.exe to choose the what to save it as.
modified on Friday, July 22, 2011 1:55 AM, to add PS
|
|
|
|
|
capital H wrote: I cannot find an option anywhere in ToDoListU.exe to choose the what to save it as. Try the menu: File / 'Save tasklist as', please.
On "the bottom" of the window you get you'll find a dropdown menu that offers:
'tasklist (*.tdl, *.xml)
and:
UNICODE tasklist (*.tdl, *.xml)
Hope this helps.
|
|
|
|
|
Found it.
Seems to be missing in 6.3.a7
I guess then my benchmark tests ANSI performance in the UNICODE build.
I will think how I can change it to UNICODE (which is not easy - interaction with dialog boxes are finicky at best)
|
|
|
|
|
capital H wrote: Seems to be missing in 6.3.a7 Yes it does, although there is a dropdown field too (but with only one option).
6.3.a8 UNICODE/ANSI is the first build of ToDOList that is working with UNICODE and ANSI lists, if I'm not mistaken.
capital H wrote: interaction with dialog boxes are finicky at best
I did it with AHK like this:
SendInput, {Altdown}{Altup}f ; it's not meant as {Altdown}f{Altup}
; I want to exclude problems with the editing controls.
SendInput, a
SendInput, {tab}
SendInput, u
SendInput, {Enter}
Works for me. Has to, since I'm not an AHK - Guru
|
|
|
|
|
I only ran it for 4500 tasks
Opening TDL+Opening tasklist is more or less equal
Saving ±5 times slower for unicode
Sorting about 50% slower for unicode (this is very surprising for me)
I am also surprised that first open is almost equal to second open time, as the tasklist that I create is very basic and ToDoList.exe needs to create a lot of new information. I guess the smaller filesize makes up for it though.
My recommendation is that optimisation (if required) should focus on the saving part. The increased filesize should increase save time by at most a factor of x2, the x5 factor must then be a result of the codepage conversions. Also saving time is more important for me than opening time, as it happens many times more.
Benchmark 4500 tasks ANSI Unicode
Opening TDL: 0.750000 0.812000
Opening TaskList: 35.766000 34.312000
Saving TaskList: 5.141000 25.188000
Sorting (10 loops): 4.265000 6.485000
Re-opening TaskList: 36.297000 36.547000
Re-saving TaskList: 5.094000 28.453000
Total time to run benchmarks: 236.437000
|
|
|
|
|