WIP: Demo: Offer Tokenizing Improvements for Highlighting #10295
Conversation
|
@JamesWTruher, I think tests could be added to
I'll work on getting this done. |
In function name and command name parsing, insure that an expandable string does not exist as these are not to be expanded. In command name, insure that redirect tokens that will be accepted as commands are flagged as such. Reference PowerShell#10275
c696360
to
a1f2ec6
Compare
|
I've included another commit to demonstrate all the potential highlighting improvements I think should be made within parsing/tokenizing. However, this also includes two changes to behavior, for |
| @@ -5284,6 +5284,13 @@ private StatementAst FunctionDeclarationRule(Token functionToken) | |||
| // A function name that matches a keyword isn't really a keyword, so don't color it that way | |||
| functionNameToken.TokenFlags &= ~TokenFlags.Keyword; | |||
|
|
|||
| // A function name is not an expandable token, so remove any nested tokens | |||
| var expandableToken = functionNameToken as StringExpandableToken; | |||
There was a problem hiding this comment.
Pattern matching makes this a little easier to read:
if (functionNameToken is StringExpandableToken expandableToken)
{
expandableToken.NestedTokens = null;
}There was a problem hiding this comment.
ahhh… that's what that suggestion message means? I had copied that from a similar part in GetCommandArgument.
There was a problem hiding this comment.
I'm not sure what suggestion you're referring to, but pattern matching was not available in C# when the code you copied was written.
There was a problem hiding this comment.
I've addressed this code in revised commit, but have a ways to go on tests.
There was a problem hiding this comment.
VS Code suggests all over 'Use Pattern Matching'. The area of the code I copied was not one such area though, probably because its too complex for the pattern matching syntax.
cee378d
to
3d83301
Compare
|
I've revised the most recent commit to address another |
Correct tokenizing output so that `--` when used as a command name does not instigate the 'stop processing parameters' function and is correctly identified as a command. Also correct further parameter tokens from being incorrectly marked as commands while in that mode. Correct tokenizing treatment of `--%` when it is a command name so as not server as the VerbatimArgumentMode, and also to be indentified as a command name token. Address tokenizing of `--` (and others) as a function name, using Token.SetIsCommandArgument. This also addresses token flags being left set when operator tokens are converted to generic tokens in Token.SetIsCommandArgument. Also addressed assumptions that TokenKind.Generic is always StringToken.
3d83301
to
6fd8322
Compare
|
This pull request has been automatically marked as Review Needed because it has been there has not been any activity for 7 days. |
In function name and command name parsing, insure that an expandable string
does not exist as these are not to be expanded.
In command name parsing, insure that redirect tokens that will be accepted as
commands are flagged as such.
In command argument processing, arguments that originate as 'Operator' tokens have their TokenFlags reverted when switched from an 'operator' TokenKind to a 'Generic' TokenKind.
In command name processing, treat
DashDashtoken as a command name and do not set in motion the 'no more parameters' mode.In command argument processing, correct the 'no more parameters' mode so additional 'parameter' tokens are not marked as command names.
In command name processing, treat
--%(verbatim argument operator) as a command and do not treat next argument verbatim.In function definitions, correct function name token of
--to be generic, and fix assumptions that generic tokens are string tokens.Reference #10275, #10347, #10348
to do
--, add tests to make sure--after a command of--still works.--%, add tests to make sure--%after a command of--%still works.PR Summary
Adjust token polishing to benefit token based highlighting.
PR Context
The syntax highlighting offered by PSReadLine has always led me through some confusion and I am sure every once in a while it confuses some others as well as to what is really accepted as syntax. However the issue is not really with PSReadLine but with the tokens that PSReadLine uses for highlighting, rather than using the AST.
It would appear that some house keeping tasks are not performed on the token collection after the AST is produced from them. Some places clean up (set indicator flags) and some places do not. The most notable item is that tokens that ultimately become
StringExpandableTokenbut when processed in to AST are treated as justStringTokenremain in that form, which then incorrectly informs any downstream applications.PR Checklist
.h,.cpp,.cs,.ps1and.psm1files have the correct copyright headerWIP:or[ WIP ]to the beginning of the title (theWIPbot will keep its status check atPendingwhile the prefix is present) and remove the prefix when the PR is ready.