|
Yeah, that's poor man's tokenizing. To be honest, .NET's regex engine is crap at it, just because they didn't do the minor work necessary to implement the feature (it's a slight "hack" or rather "twist" on a a|b|c such that each expression a,b, and c has a symbol id associated with it, and the engine will tell you which it matched. It goes through the text beginning to end, reporting all matches like that, one row in the table for each match..
Your input spec might look something like this (.rl format)
VerbatimStringLiteral= '@"([^"]|"")*"'
StringLiteral='"([^"]|\\.)*"'
CharacterLiteral= '[\']([^\']|\\.)([\'])'
IntegerLiteral= '(0x[0-9A-Fa-f]{1,16}|([0-9]+))([Uu][Ll]?|[Ll][Uu]?)?'
FloatLiteral= '(([0-9]+)(\.[0-9]+)?([Ee][+-]?[0-9]+)?[DdMmFf]?)|((\.[0-9]+)([Ee][+-]?[0-9]+)?[DdMmFf]?)'
// the following takes a long time to generate
//Keyword = 'abstract|as|base|bool|break|byte|case|catch|char|checked|class|const|continue|decimal|default|delegate|do|double|else|enum|event|explicit|extern|false|finally|fixed|float|for|foreach|goto|if|implicit|in|int|interface|internal|is|lock|long|namespace|new|null|object|operator|out|override|params|private|protected|public|readonly|ref|return|sbyte|sealed|short|sizeof|stackalloc|static|string|struct|switch|this|throw|true|try|typeof|uint|ulong|unchecked|unsafe|ushort|using|virtual|void|volatile|while'
Whitespace<hidden>='[\t\r\n\v\f ]+'
Identifier='[_[:IsLetter:]][_[:IsLetterOrDigit:]]*'
CommentBlock<id=40,blockEnd="*/">="/*"
//Bar="bar"
Forgive the word wrapping but it's a line based grammar.
So if you tokenize something something by calling Tokenize you get back a row for each match and what Symbol it was plus where it was in the document and its actual value (like CommentBlock at position 3, value "/* bar */") and you get many of those for a potential string.
Real programmers use butterflies
|
|
|
|
|
I am working on a Kotlin app that needs to load a large file at startup. Because it can take up to 20 seconds it must be done in a separate thread so it won't block the UI thread. When it is loaded I need to send a signal to the main UI thread, so that the main thread will know the file is ready for further processing.
Being a Kotlin greenhorn, I battled for hours to send a signal from the background thread to the main thread, with no luck. Then I remembered reading something about a built-in Kotlin method :
runOnUiThread() . "Surely it cannot be that simple", I thought. But in desperation I tried adding this statement to the end of the background thread:
runOnUiThread { someMethodOnUiThread() }And voila! it worked. Apparently this little gem inserts the requested method into the execution queue of the UI thread.
Sometimes we look for complicated solutions when a perfect simple solution is right under our noses.
By the way: Please don't tell me I should have used a Kotlin coroutine. I tried that, but when you use a file reading class like ObjectInputStream the coroutine starts blocking the UI thread.
Get me coffee and no one gets hurt!
modified 30-Oct-21 15:09pm.
|
|
|
|
|
I thought that's what postMessage does - posts a message onto the UI thread.
Window.postMessage() - Web APIs | MDN[^]
But then again, I don't write apps in Kotlin so I needed something more native, using the Worker class Using Web Workers - Web APIs | MDN[^] and various machinations to deal with the fact that I didn't want external files for the source for the worker. 
|
|
|
|
|
Thanks. I may just try that for fun!
Get me coffee and no one gets hurt!
|
|
|
|
|
Quote: coroutine starts blocking the UI thread
I believe that you can set the dispatcher context of the coroutine thread so that it runs on a thread other than the UI thread.
Dispatchers in Kotlin Coroutines - GeeksforGeeks[^]
That said if your solution works then that's fine as far as I am concerned(I am from the school of software needs to work rather than it needs to be beautifully architected but not work) - Kotlin still seems to have a few quirks with coroutines.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
My experience with coroutines is that they run in non-blocking mode, until you start doing disk io. Then they block the UI thread. I have also seen a number of posts confirming this behavior. I think it was on Stackoverflow.
Get me coffee and no one gets hurt!
|
|
|
|
|
 go with what works.
I would have thought that disk IO would occupy the IO thread but as you hint coroutines can be a bit weird.
“That which can be asserted without evidence, can be dismissed without evidence.”
― Christopher Hitchens
|
|
|
|
|
*sigh* - (half serious here) what even is the point of UTF16? You still have the possibility of encountering surrogate characters, which means if you want to support unicode streams you have to handle that possibility.
MSSQL supports UTF16, but not UTF32
Consequently, here's me fetching the next character of an NVARCHAR(x) or NTEXT stream, with UTF32 support. Forgive the grotty code, as there is no unsigned data types and no bit shifts, etc
DECLARE @valueEnd INT = DATALENGTH(@value)/2+1
DECLARE @index INT = 1
DECLARE @ch BIGINT
DECLARE @tch BIGINT
...
SET @ch = UNICODE(SUBSTRING(@value,@index,1))
SET @tch = @ch - 0xd800
IF @tch < 0 SET @tch = @tch + 2147483648
IF @tch < 2048
BEGIN
SET @ch = @ch * 1024
SET @index = @index + 1
IF @index >= @valueEnd RETURN -1
SET @ch = @ch + UNICODE(SUBSTRING(@value,@index,1)) - 0x35fdc00
END
This is hateful and slow. I haven't even tested it yet. I know it doesn't gracefully handle any and all invalid unicode streams but to make it do that is even worse. To heck with this.
Why UTF16 with no functions to convert a surrogate pair to UTF32? it's ridic.
Someone asked me the other day why I don't like Microsoft SQL Server. Here's reason #1359
Bad MSSQL! BAD!
Real programmers use butterflies
|
|
|
|
|
Yep it is something like a compromise. UTF16 is usually enough but we also need to do the unusual 
Surrogate characters is only one beast.
Expressing accented characters e.g. 'é' in different ways is one other. Ok, here at least we have the possibility to normalize it (https://stackoverflow.com/questions/7811976/normalize-unicode-string-in-sql-server).
modified 27-Nov-21 21:01pm.
|
|
|
|
|
Isn't an issue for me, as this is a sqlized regex engine.
All matching conditions are specified using UTF32 stored as BigInt values (since SQL has no unsigned ints)
All incoming characters are resolved using the code I posted.
Accent marks will be matched properly.
Displaying them is firmly in "Somebody else's problem" territory.
Real programmers use butterflies
|
|
|
|
|
UTF-16 is the result of developers underestimating how many characters would be needed for a universal encoding and lingers like the stench of a soiled diaper because MS rushed it into production in the 90s before anyone realized it was a mistake. 
Did you ever see history portrayed as an old man with a wise brow and pulseless heart, weighing all things in the balance of reason?
Is not rather the genius of history like an eternal, imploring maiden, full of fire, with a burning heart and flaming soul, humanly warm and humanly beautiful?
--Zachris Topelius
|
|
|
|
|
Just a vent. 30 minutes plus wasted, my guess is an MS update, although I didn't think they updated on Fridays.
Wacom tablet (old) was working just fine this morning. Doing a lot of work in Blender. 30 minutes ago (+) I went to use it and the pen totally stopped working in Blender after starting it from scratch. But the pen worked as usual on the desktop, so it was communicating with the OS.
It was one of those WTH??? puzzles. I feel sorry for anyone who faces such a situation who is not a good troubleshooter. I feel sorry for myself, because of the headache!
Clicked around and deleted and reinstalled the Blender options in the Wacom program. Didn't work. Finally found that Wacom had released an update for their drivers. So out of curiosity and prayer, downloaded it and installed. Everything started working again.
Why would the OS stop sending signals to specific programs when a third party updates their driver? Did MS send some update that triggered the situation, 'cause unless Wacom has a time bomb in their code, nothing else makes sense. (Unless I have a virus, but my antivirus says no, and it stared working with the updated drivers.)
Grrrr..... F whoever is responsible for such brain-dead decisions, or coding!
PS - anyone know why my laptop monitor brightness would change when Blender becomes active? I've got a 24" monitor set up as the primary screen, and when I open and close Blender the laptop brightness changes (not the external screen). I've disabled Game Mode, and also killed the Dell display program, so I don't know what else it could be. Thought the two items were related, but they don't appear to be.
modified 29-Oct-21 20:31pm.
|
|
|
|
|
Hi,
David O'Neil wrote: Why would the OS stop sending signals to specific programs when a third party updates their driver?
Blender is using DirectX which probably means that it's using Raw Input[^] and opening your HID devices with exclusive device access mode[^]. Exclusive access will fail if another program already has exclusive access. Most DirectX software (such as video games) do this.
David O'Neil wrote: PS - anyone know why my laptop monitor brightness would change when Blender becomes active?
Just a guess here... but Blender could be setting the ICC color profile[^] and doing calibration. It could also be a third-party library doing that, I think Blender uses an open source library OpenColorIO [^] for color management. Maybe you can ask about it on their Github issue forum.
The events that occur on your workstation aren't always caused by an operating system update.
Best Wishes,
-David Delaune
|
|
|
|
|
Randor wrote: The events that occur on your workstation aren't always caused by an operating system update.
I did not update Blender, the Wacom drivers, or do a manual Windows update, but Blender's response changed. The only realistic option I see is that a Windows update changed how it interacted with Blender. It may be related to the Raw Input you pointed out, but it only started working again after I updated Wacom's drivers. Unless Wacom has a time bomb check in their drivers that stop certain interactions under certain conditions.
Randor wrote: Just a guess here... but Blender could be setting the ICC color profile[^]
You could very well be right.
Thanks for the input, and have a great weekend!
|
|
|
|
|
PS - I even rebooted Windows and tried running Blender only, and the shenanigans happened.
|
|
|
|
|
David O'Neil wrote: Grrrr..... F whoever is responsible for such brain-dead decisions, or coding!
(Putting on my Mickey Mouse hat) M-I-C-K-E-Y-S-O-F-T
I have lived with several Zen masters - all of them were cats.
His last invention was an evil Lasagna. It didn't kill anyone, and it actually tasted pretty good.
|
|
|
|
|
Only 30 minutes?
Come back when you've wasted days on some stupid issue 
Any issue that didn't take you at least hours isn't even worth mentioning 
|
|
|
|
|
Sander Rossel wrote: Come back when you've wasted days
Been there, done that. That is probably why this only took 30 plus minutes. Once called OpenDialogBox (or whatever function it was) with one wrong argument. Felt really stupid that time. Another time had different compilation options for a DLL and the main project due to macros. That was a real head scratcher. But can diagnose that real quick, now!
|
|
|
|
|
After five years of working for them, and 15 years of doing work for them, I've finally ended my working relationship with Microsoft. I'm technically retired*.
Now I can finally tell you all just how I feel about them.
Narrator's voice - He's always done that: they're great, but make some really stupid decisions at times.
*shane, Bob, and I will still be doing the newsletter, so I don't get off that easily.
TTFN - Kent
|
|
|
|
|
"but make some really stupid decisions at times."
Yes, I've suffered and continue to suffer from their stupid decisions.
Charlie Gilley
<italic>Stuck in a dysfunctional matrix from which I must escape...
"Where liberty dwells, there is my country." B. Franklin, 1783
“They who can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety.” BF, 1759
|
|
|
|
|
Kent Sharkey wrote: I'm technically retired*.
Yeah, I used that excuse before.
Just stay a little while?
Bastard Programmer from Hell 
"If you just follow the bacon Eddy, wherever it leads you, then you won't have to think about politics." -- Some Bell.
|
|
|
|
|
while (jobs--) { doNext(); } enjoy();What a nice piece of code. Did you try to make it a template?
I'm happy you'll keep sharing your insider views; you're responsible of so many coffee-spillings-on-my-shirt
"Five fruits and vegetables a day? What a joke!
Personally, after the third watermelon, I'm full."
|
|
|
|
|
phil.o wrote: I'm happy you'll keep sharing your insider views; you're responsible of so many coffee-spillings-on-my-shirt
+1
M.D.V.
If something has a solution... Why do we have to worry about?. If it has no solution... For what reason do we have to worry about?
Help me to understand what I'm saying, and I'll explain it better to you
Rating helpful answers is nice, but saying thanks can be even nicer.
|
|
|
|
|
Congrats!
Is this your plan for retirement life: travel around the world?
diligent hands rule....
|
|
|
|
|
You are not technically retired until all your friends, relative and neighbors get rid of their computers.
I can't retire fully until my kids retire.
Lou
>64
If you can keep your head while those about you are losing theirs, perhaps you don't understand the situation.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
