|
An example: I create a list of #2880 'PointF instances which represent points on a hypothetical circle with a radius of #1000 units (360 degrees "sampled" at a "frequency" of 0.125d). I serialize this List in compressed form to a file, and write code to access it, and de-serialize it, on demand. All this, easy to do.
Then, as I want to use a set of points for a circle of a different size radius, I can open the file, de-serialize to a new List<PointF> ... no problem. And, of course, I can then "down-sample" that data to create a smaller list of PointF, that would, in effect, give me data for a "smaller" circle.
If I had a set of data too large for available memory, I could, of course, do a a file read where I down-sample by some factor, importing only the data set whose size would be usable given available memory.
But, assuming the whole-banana can be de-serialized into available memory, what's the best practice (faster, less memory use) for "transforming" the "master list of PointF" ... where that transform may involve both scaling and translation ... but, let's exclude rotation.
Possible strategies ?
1. foreach loop, and apply translation and scale per PointF
2. use a Matrix to achieve transform. Using this code now:
private static PointF[] TransformPtF(List<PointF> pts, Matrix mtrx)
{
PointF[] result = new PointF[pts.Count];
pts.CopyTo(result);
mtrx.TransformPoints(result);
return result;
}3. use 'unsafe code ... perhaps requiring "master data" to be saved in a different format ? byte-array ?
4. use Linq 'ConvertAll with a custom converter ?
5. or ?
Appreciate your thoughts.
thanks, Bill
«In art as in science there is no delight without the detail ... Let me repeat that unless these are thoroughly understood and remembered, all “general ideas” (so easily acquired, so profitably resold) must necessarily remain but worn passports allowing their bearers short cuts from one area of ignorance to another.» Vladimir Nabokov, commentary on translation of “Eugene Onegin.”
modified 21-Feb-16 0:40am.
|
|
|
|
|
I'm not sure wether this will help, but, have you considered a memory-mapped file for accessing your data. That would give you speed to do any sort of processing (possibly more aligned with your point #1) while the system 'pages' the data to and from memory and the file on disk - transparent to you .. maybe this is "5. or ?"
|
|
|
|
|
edit: there's some code here that's suggestive: [^] : perhaps binary serializing the PointF data as one big Byte Array is a strategy, then reading it into a memory stream, is a strategy.
Thanks, Garth, I'll look for an example of that. And, I haven't explore the idea of de-serialzing the "master list" of PointF to a memory stream and working with that ... not sure if I could somehow use Matrix.Tranform there.
«In art as in science there is no delight without the detail ... Let me repeat that unless these are thoroughly understood and remembered, all “general ideas” (so easily acquired, so profitably resold) must necessarily remain but worn passports allowing their bearers short cuts from one area of ignorance to another.» Vladimir Nabokov, commentary on translation of “Eugene Onegin.”
modified 21-Feb-16 2:11am.
|
|
|
|
|
When it comes to situations where I need to play with streams of data, I tend to go to Rx as my default grab bag of tools. Basically, Rx (Reactive eXtensions) is a very clever implementation of the Observable pattern that allows you to work with a Linq style syntax and do operations such as throttling, transformation and so on.
This space for rent
|
|
|
|
|
This should probably be a separate question, but I often find myself in a position where I want to change the threads of the subroutine that is called.
For instance, say I want the points created by MouseMove it's quite easy to get them observing on the UIThread:
var Moveing = Observable.FromEventPattern<MouseEventArgs>(this, "MouseMove").Select(evt => evt.EventArgs.GetPosition(this));
Moveing.Subscribe(evt => {
this.Title = evt.ToString();
});
But after 1 minute assume that I want to calcualte some long running prossess off the UIThread, like the avarage posision of the mouse for instance, and when completed post the result back to the UIThread. The ObserveOn or SubscribeOn didnt quite seem to fit, as I do already ObserveOn the UI thread in this case. I could use the method SubscribeOn but that seemed to imply that the subscribed code ran in a different thread, and I coudnt really post the result on the UI thread.
I was hoping that the Rx would make this so easy for me, but it's giving me some headaches insted.
|
|
|
|
|
When I get some time free, I'll attempt to pull something together for you to demonstrate this in operation.
This space for rent
|
|
|
|
|
Hi Bill,
1.
From your list, given such a task, I would instinctively go for TransformPoints(), although I haven't ever used it.
2.
However.
Lacking contextual info, it is unclear to me why you would have a PointF[] array in the first place. A PointF[] may of course be a popular parameter type in the API's you are using, if not I would think twice before introducing it.
2a.
If this is a graphical WinForms application, surely PointF[] are sometimes useful. However chances are the transformations provided by the Graphics class could do the job you need done, unless drastic downscaling would cause way too many irrelevant points to be drawn.
2b.
"All" the points of a circle can be generated rather cheaply by algorithmic rotation, using something like:
xNext = x*cos(alpha) - y*sin(alpha);
yNext = y*cos(alpha) + x*sin(alpha);
where alpha is the angular step you want to use to step the circle's circumference. Obviously you need to calculate cos(alpha) and sin(alpha) only once, so each point would cost 4 muls + 2 adds, to be weighed against disk accesses and memory usage.
2b.PS: yes, this iterative approach will accumulate rounding errors; it may well be acceptable for a few hundred/thousand points on your circle. And using double rather than float probably wouldn't increase the cost at all.
2c.
A circle being extremely symmetrical, if you decide to store circular points anyhow, you could consider only storing one eight of a circle (say everything from 0 to 45 degrees). Then, having separate x[] and y[] arrays may prove easier than a single PointF[].
2c.PS: and when you store one eight of a circle, you should consider dealing with the eight points that belong together in a single go, before stepping to the next 8-tuple.
Hope any of this is relevant to you.
modified 21-Feb-16 18:13pm.
|
|
|
|
|
I'm using this class[^] to handle file/folder changes.
When a file is dropped into a folder, the Created event occurs right away - even though Windows is still copying the file. I suspect that the .Net FileSystemWatcher, which this class extends, will probably do the same.
Anyone know a solution for this? I don't want the FileSystemWatcher to report events until the file is completely copied into the watched folder.
Thanks
If it's not broken, fix it until it is
|
|
|
|
|
|
I wonder if comparing the file's Created DateTime against the DateTime the event was called would do it. Maybe the file's DateTime isn't set until the copy is complete.
I'll have to try it
If it's not broken, fix it until it is
|
|
|
|
|
I know that LastWriteTime gets updated at file close, but I am not sure if it also gets updated on each write. Should not be too difficult to test.
|
|
|
|
|
|
I saw the first two, but not the last.
I need to spend more time reading before keyboarding again
Thanks
If it's not broken, fix it until it is
|
|
|
|
|
what I tend to do when awaiting a large file is send two files, the one you are interested in, followed by a very small one (a sentinel) for which I order a FileSystemWatcher. Don't rely on file datetime stuff, as it depends on the actual file system, which could be a USB stick (often FAT), a network disk (might be UNIX based), etc.
|
|
|
|
|
Agreed. I've seen this approach used very successfully in several production scenarios across various systems.
|
|
|
|
|
yup - used to call this technique 'sentinel file' - very handy for processing a large data file being FTP'd to you from a mainframe, you get the mainframe to send you a small sentinel file and wait for that to kick off further processing
|
|
|
|
|
As you'll see, there are all sorts of solutions, and as JSOP points out (iirc), the notifications you get depend on the application thats creating/? the file
One strategy I used (when not using 'Sentinel Files' as per my comment to Luc), was to queue a notification from FileWatcher, and then periodically see if I could get exclusive access to the file and/or combine the notification with a dir list check - if so, she's good to go, if not, wait a period - not sure if JSOP outlined that sort of strategy
|
|
|
|
|
When .Length in the file's FileInfo is greater than 0 (and .Exists), the file has "arrived".
|
|
|
|
|
The file's size would be greater than 0 the moment Windows starts writing to it, and Exists would be true... But the file could still be in the process of being written.
If it's not broken, fix it until it is
|
|
|
|
|
|
Yes
The Size is the size of the file being copied, and it exists
If it's not broken, fix it until it is
|
|
|
|
|
Me too now ... Changed since I first used it 4 (?) years ago.
Did notice something else this time though (.NET 4.6):
Fires 3 events during the download:
1) Created
2) Changed
3) Changed
First 2 fired as soon as the download started; the last one (#3), corresponded to the download being complete.
|
|
|
|
|
Ya but the problem is that you never really know what event are going to fire. Different apps cause difference events to fire.
I'll probably do this in the Watcher events:
private FileWatcher_Changed(...)
{
while(!IsFileAvailable())
{
}
}
public bool IsFileAvailable(String sFilename)
{
bool results = false;
try
{
using (var inputStream = File.Open(sFilename, FileMode.Open, FileAccess.Read, FileShare.None))
{
results = inputStream.Length > 0;
}
}
catch (Exception)
{
results = false;
}
return results;
}
[UPDATE]
Just tried it. Worked fine.
If it's not broken, fix it until it is
|
|
|
|
|
I avoid tying up the file watcher by queueing the events and kicking off a background worker (if one isn't already running) to deal with the queued events. Better than polling (IMO).
|
|
|
|
|
Meaning each time the Changed is called you fire off a BG worker wait for the file to be done?
Makes sense.. Then multiple files can be handles at one time.
If it's not broken, fix it until it is
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
