Welcome to the Lounge

Duckduckgo had the link to the article as the second search result, google had the link to the article on page 14.

Chances are that you've made many FSTs without calling them as such. At least in electronics, it is rare to have a FSA evolve from state to state without producing any output. The set of signals that you triggered during transitions are the output tape of a FST.

Quote:

the set of signals that you triggered during transitions are the output tape of a FST" - are you saying that I write out each transition?

Quote:

However, the output only gets emitted on Accept.

Quote:

At that point I yield the input I've received so far, which is not exactly (forensically speaking) a map of my transitions, but rather what the transitions gathered from the input tape, so close, but not quite the same thing. (I could modify it to record state transitions easily but I don't understand why I'd do that.)

FA wspace = FA.Parse("[ \\t\\r\\n]+");
FA ident = FA.Parse("[A-Za-z_][A-Za-z0-9_]*");
FA num= FA.Parse(@"(0|-?([1-9][0-9]*))((\.[0-9]+[Ee]\-?[1-9][0-9]*)?|\.[0-9]+)");

FA lexer = FA.ToLexer(new FA[] { wspace, ident, num });
foreach(FAMatch match in lexer.Run("foo 100 bar baz )#($")) {
    Console.Write("{0} ",match.SymbolId);
}

// produces 
// 1 0 2 0 1 0 1 0 -1

Quote:

It's interesting because you basically suggested to me (if I understand you), that a lexer/tokenizer is essentially an FST.

Quote:

FSTs are used in the lexical analysis phase of compilers to associate semantic value with the discovered tokens.

Quote:

My faith! For more than forty years I have been speaking prose while knowing nothing of it, and I am the most obliged person in the world to you for telling me so.