As an example of the kind of thing we're thinking about:

#3121 - shell-driven autocomplete menus that can provide rich info to the user, outside the limitations of the text grid itself. Sure, there are shells that implement similar features, but with the caveat that they need to print the menu to the buffer themselves. This leads to a lot of different implementations across different shells. Having the terminal display the suggestions also allows the terminal to provide semantic context to screen readers.

I really don't want this to be something that's Windows Terminal-specific, or based on heuristics. Shells have pretty great auto-completion that can already be configured. So I'd rather let the commandline app do the driving. Imagine if you could have vim/emacs/whoever provide the completeopt menu but in the terminal directly. Whenever {the next hot CLI editor} comes around, they could just leverage this autocomplete menu from the Terminal instead of having to roll all that rendering themselves. By making it shell-driven, it could be something any terminal emulator could also support.

Related to this, having some mechanism of discoverability for context-based documentation -- this would go a long way for helping on-board newer users to command-line based system. One could envision something like support for a Dash/Velocity-style documentation that would switch based on the context that the terminal is in (e.g., zsh/bash/ps, moving over to a different context for an ssh-session or screen, or even interactive environments like ipython). Something that would enable quick discoverability of what options one can/should use without having to switch back and forth to man pages.

@mubdi I actually really like that. Like, if the terminal knew what the commandline was being typed, it wouldn't be that hard to hook that up to manpages or Velocity
or whatever for that commandline. Those docs could totally slide in as a pane side-by-side with the terminal output.

BOY would @cinnamon-msft love to hear this. I think we had some discussion about a theme marketplace in the past. There's really great potential for some of this with the fragment extensions that we've already got, but there's some pretty mighty overhead between authoring a color scheme and publishing it in the Microsoft Store. There's gotta be an easier way.

I really think just a domain we own would be easier than the Store. Could even do a GitHub website?

Oh for sure. It's JSON blobs. How hard could that be to manage?

if you go for the domain you own, perhaps you could use a GitHub Action to validate syntax then combine a directory of /themes into an all-themes.json then boop, put that on GH Pages with each merge to master 🥳

Wouldn't these be things better suited to the shell, not the terminal? Like, PowerShell already has support for MenuComplete for a "graphical" autocomplete.

Maybe I'm missing something?

Autocomplete is offered by some other terminals as well -- it's one step towards lifting the behavior up out of the shell. It's pretty compelling, thanks!

For example, this is what I see in Prompt for iOS:

dang now how are they doing that? There's bash-completions sure, but you think that's a "we actually know what the commandline you've typed is and are doing the completion ourselves"

I suppose theoretically you could modify readline to send the autocomplete sequence to the terminal

If I had to guess I'd say they literally watch what you type, delimited by Enter, and tokenize it.

Depending on your shell, you can probably already do that. In cmd.exe and PowerShell, for example, ctrl+arrows will move the cursor a whole word at a time.

I don't think the value proposition of terminal based interfaces is in their usage of the keyboard per se, but in the simplicity of writing interfaces between programs and the terminal (as simple as command-line arguments or as complex as zork or nethack), and the simplicity of interacting with those interfaces. In a lot of ways, this proposal could add friction to that simplicity. I don't discount that concern. But I think that if your shell is something like xonsh or powershell (or maybe something based on babshka), then you're scripting using objects, not bytestreams. And these sorts of shells seem to work rather well. Objects have metadata, that tells you what it is, and because the shells have the sophistication to handle that, why shouldn't our terminals have the sophistication to make use of it? It just, setting aside the technical limitations for a minute it just doesn't make sense to me why I can't have the simple interface of typing in what I want the computer to do, and also of clicking on a thing when it's right there on the screen.

I suppose part of the trick then is how do we get these objects that have some internal representation to the terminal in a language-agnostic manner? For whatever reason, I can't think about this in any other way than "well duh, just serialize the objects to {something like JSON} and then write that to the terminal as a string of text" and now we haven't really made any progress.

Heck, powershell still converts the objects to text when you write-host.

<going way further afield>

Maybe we've gotta let there be a "fourth pipe" for transmitting richer data. Imagine (the horror of) writing a COM object to that fourth pipe, and then the Terminal on the other side can say "Okay, what are you? How would you like to be displayed?"

Of course, that object would have to stay alive in memory somewhere. Even ignoring something like ssh. When you're done executing ls, ls is gone - the process is totally lost. any objects it created would be gone too.

Well, that's just it. ls may be gone, but it's output is not. That's the trick maybe.

The object conversion might need to be object rendering (to some kind of serializable representation the terminal understands), where the shell keeps the object alive while it's in visible history, but I think that's the shell's problem. The shell takes the objects, and sends terminal something like

$> ls

"""
^~{"render-value" : "<name-of-directory>",  
     "reference" : "<reference the shell understands>"
     "type" : "directory", 
     "full-path" : "<full-path>",
     "on-click" : "<stuff the terminal understands>", 
     "on-double-click" : "<shell> <command to rename directory>"}

^~{"render-value" : "<name-of-file>",  
     "type" : "jpg", 
     "reference" : "<reference the shell understands>",
     "full-path" : "<full-path>",
     "on-click" : "<terminal command to append <visual-depiction> to render-value>", 
     "on-double-click" : "<terminal command to display image in new-buffer>"}\n
"""

which displays as:

$> ls
<name-of-directory>
<name-of-file>

For an editable object, the shell would need to provide an on-edit.

To your point, I think that having an escape character(s) to tell the terminal "this is an object referring to a shell object" might be sufficient. Copying a thing from the terminal, might be copying the reference of the thing instead, which I guess would be an issue if your other programs don't understand that reference. But I think for most things, you don't really need to have the objects live, just some kind of reference for what it is on the filesystem or something that the shell can use to get a fuller representation of the thing, and, for that, simple, portable representations like paths will do. And if you do need a reference to a live object, you're probably copy-pasting right back into the terminal anyway, so it's a non-issue.

you folks seem to not know about "nushell" because it already does a lot of this, if not all of it.

https://www.nushell.sh/
https://github.com/nushell/nushell

Also, shells really shouldn't have to be purely text based. You should be able to go 'cat cat.jpg' and get a picture of a cat in a terminal.

There's been https://github.com/hackerb9/lsix. But WT needs to support Sixel first.

(Apologies for the redundancy, missed that this would likely fall under Sixel support.)

For sure. This is just another riff on "images in the terminal", let it be sixel(#448), the kitty protocol, or some other good protocol that hasn't been standardized yet.

after reading the "some other good protocol" link, i'm on board with abandoning support for sixel and supporting something like that. if it matters.

Already on the backlog 😉 #3459

Can you please explain how copying as SVG would be useful? I don't understand the utility of that.

@MathiasMagnus Something like #1084 maybe?

@zadjii-msft Something like 1084, yes. @naikrovek My main use case is copying stuff from the terminal to locations where I want to keep the original formatting but keep layout and not make it subject to word-wrapping and other formatting butchering but not turn it into raster. This comes up in two places:

• Making PowerPoint slides, but don't want to turn the presentation into a 10MB slidedeck, only because I got formatted console text on every slide.
• Copying very narrow but long parts of text (like one entire line) typically looks terrible in raster but is fine in vector. I'm sure you've seen thumbnails of PNGs on twitter where a 1600*40 pixel image is inserted.

In my opinion the ways one would want to copy console content is either:

• As plain text ✔
• As HTML5 ✔
• As raster ✔ (Snip tool or terminal capability)
• As vector ❌ (Absolutely no way to do it today)

All of these methods have pros/cons, and whenever I'm creating uni material, I often find looking for the last missing option. Having come across the tweet I linked is the exact kind of thing I'd like to embed into materials without losing quality or forcing DPI of my screen onto consumers.

Do let me know if you think it's nonsense or if I'm missing the forest for the trees.

I love this stuff.

timestamped lines

this does seem like something the client app could do, but might be handy to let the terminal just do it for you. Looks like we've already got an issue for this: #9331.

stdout/stderr coloring

Maybe the hardest one. At least in windows, there's actually no internal difference between out and err. They both get hooked up to the same console backend, and they get hooked up without any indication to the console that they were out or err. That one's a thinker - how could we create multiple consoles, different ones for out and err, and interleave the output of the two streams in the correct order? Maybe that's not the right way to go about it.

quick way to see only lines that match a regexp

Oh interesting. Almost like the inverse of highlighting all the matches - instead, fade everything that doesn't match. I like it.

Programs can output abbreviations -- mouseover to see expansion

This one I don't think I understand - can you maybe give some more details? this seems like a really great idea, but I wanna make sure I'm imagining the same thing as you.

Quick and easy bookmarks

You're gonna love my current passion project #11000

Indicator of waiting on computation

You're gonna like #6700, #7955 et. al.

Thanks, I'm excited to see some of these under way already!

Almost like the inverse of highlighting all the matches - instead, fade everything that doesn't match.

Actually, I was thinking of completely hiding non-matching lines and closing the gaps. Maybe it would be in a separate window for clarity. Another way of explaining it, Unix-style, is /some/program | tee buffer.txt, the program ends, then later, grep someregexp buffer.txt -- but without having to manage buffer.txt manually and without having to have planned to do this beforehand. Maybe this fits in with the notebook-style ideas (though I'm not very familiar or comfortable with notebooks) -- you collected your output into a notebook cell, then later on do some more processing on that cell's contents, without necessarily altering the original cell contents. But it would be nice to do with the entire available scrollback buffer too.

Programs can output abbreviations -- mouseover to see expansion

Ok. The example that is coming to mind is fish shell's abbreviation of the working directory (at least in my configuration). The prompt might say ~/t/o/o/utils for the path /home/scintill/tmp/offlineimap-6.5.7/offlineimap/utils. I like the abbreviation, but it would also be handy to be able to hover over that (or other mode of interaction) and see what the full path was, especially if it was some time ago and you can't easily just pwd or such. This may not be a very interesting example, but the general feature could have lots of uses. Maybe it could be extended to blocks of text, such that an entire multi-line block could be abbreviated one way but be expandable/collapsible with more details available. The application would output both the abbreviation and the full text, annotated with control codes, and the terminal would handle expanding.

@crutkas 👀

@DHowett actually had this crazy prototype a while back that did exactly this! It would capture all the API calls to the console with the timing, so you could replay the exact set of API calls later. It was really cool, I wonder where that went.

idly: I wonder if the API calls would still work for different schemes - probably? I'm guessing that since the conpty's font and color table don't affect the Terminal's side, that it would work as you'd expect.

Apart from the conhost dump/replay work (which happens on an API transaction level and is architecture/bitness-specific), I would love for us to get Asciinema support. They have a well-documented standard format for this, and a web-based player that can be themed and all.

Asciinema is fine, I just want a way to replay something the way it showed up on my screen, without necessarily recording it as a video.

Supporting Asciinema replay files would work, I'm sure, and I would PREFER not to need to visit a webpage to play the file, so adding support to Terminal for that would be lovely. Maybe not in scope, or is much less feasible that using the existing player.

Honestly, I'll take what I can get. 🙂

is there a sandbox we can use to author these shaders and see them in real time, like shadertoy?

Not really 😕 Some time ago I had a branch where the toggleShadeEffects action would "warm-reload" the shader, reparsing it each time shader effects got enabled. I don't think I ever got that shipped though.

I have a problem, could be that I have an error in the declaration because terminal sais me "Selected shader can't be found"

"experimental.retroTerminalEffect": true,
"experimental.pixelShaderPath": "Retro.h1s1",

This is a piece of JSON, but I'm not sure if it's fine from my part. I've test deleting the first line "experimental.retroTerminalEffect": true, an leaving only the second "experimental.pixelShaderPath": "Retro.h1s1", but still de same, so, apparently I'm doing it bad, how I should do?

And other thing, you said that I could choose the shader effect that I wants, where ir the list of available shaders?

Thanks a lot an congrats for your job

make sure the file extension of your shader and the file extension of the specified shader file name in the configuration match. (I don't think h1s1 is correct. it should be hlsl.)

Shaders are here.

Thats perfect, was exactly that (problem with my browser font), great, in additon I modified some params of .hlsl and know looks as I want.
Thanks for the help

I'm not sure that the existing solutions are going to work, especially on high-dpi displays. If we drop the need for the pixel-perfect display of images then there's room to grow. Pixel perfect image display can happen in a terminal, but it should not be a requirement, and I believe that Kitty, iTerm2, and Sixel all assume a pixel-perfect rendering.

I am willing to get away from the existing solutions if a new solution is demonstrably better. I think we spend too much effort accommodating backwards compatibility, in general. It's ok to move forward.

Kitty is literally the best way forward. It's the most advanced protocol and provides the most support. It's also the most complicated one to implement which is probably why most terminals aren't adopting it yet. Wezterm has though so there's hope others will too. If there's something Kitty is missing that needs addressing, just raise an issue with kovid. I'm sure he'd make a change if it made sense (he's already done so before at request of the notcurses author). The docs are very well defined for implementing. It's just a matter of time and resources. Same can be said for the other 2.

There's some deep history rooted in why pixel perfection is a must in the terminal. See the spec for the "Good Image Protocol" in the terminal working group. Too many things can and will go wrong without it. This isn't some youtube video we're talking about, we're talking about a terminal graphics is just one thing you'd be viewing. You may still want to issue commands or scroll the history buffer up to view previous entries/results. There's alot going on in the background to consider everything a user can possibly do.

There's some deep history rooted in why pixel perfection is a must in the terminal.

I don't see how pixel perfect images can be done without compromise, somewhere. Some things that come to mind immediately:

• the dimensions, in pixels, of the displayed font (which determine the segment size of the image to display in a character's position)
• the user zooming in or out at any time (changing the size of the character cell at an arbitrary time)
• the user changing the font or font size at any time (same)
• the user changing the size of the window at any time (changing the extents of valid display area)
• what if the image is too wide to display in the terminal? do you wrap?

are you planning to ALWAYS display images pixel perfectly? what about in high-dpi environments, where the display scaling is 200%? what about 300%? at some point, the image is tiny and the text surrounding it is huge in comparison.

I don't see how one can do this while maintaining pixel-to-pixel perfection at all. It just doesn't make sense.

pixel perfect display of images should be possible, but never mandated.

FWIW, I have a work in progress Sixel implementation for Windows Terminal which most definitely does not use pixel-perfect rendering. It always renders at a cell size of 10x20 to match the behavior of a real VT340. And as you say, there are a whole lot of reasons why that makes sense.

If it ever got far enough along to be merged, though, I would probably add an option for users to choose whether they wanted pixel-perfect rendering or not, because I know some people are quite keen on that. But even then the cell resolution would be locked in at startup to avoid all of the problems that come with runtime font changes.

oh, and make it an open spec. ncurses and its ilk need to die, already.

You mean something like CSI 14 t or TIOCGWINSZ ioctl? There's a couple issues open requesting those. While ncurses itself is pretty old, there are some modern alternatives. In particular while ncurses is used for everything not directly supported, notcurses is a pretty good TUI library and can also be used for non-TUI's too.

I want a terminal control set extension or mode where I can:

1. define a thing and provide its contents
2. display the contents of that thing in a rectangle with a given location and size
3. use my mouse to hover over that rectangle and scroll the content without fetching new data from the remote computer/application

the nature of things like ... the entire history of terminals, which ncurses relies upon, preclude a solution like this.

I'm saying that I want something smarter than a dumb terminal, backwards compatibility be damned once you're in this new mode.

I want a text-only, bandwidth friendly way to use content-dense application user interfaces in the terminal. I don't want to fetch the entire screen every time anything scrolls into view.

Huh, not sure how much of this I can really comment on currently. This thread was intended as a "10 year" view, and that's what we got. That being said, we really only plan out a few months at a time - priorities seem to shift faster than that, so planning any farther than that is usually an exercise in futility.

There's basically a few camps that these responses were grouped into:

• Images in the Terminal
• New Terminal features
• Richer CLI application output

The first is pretty well understood problem, one that will just take a while to implement. I don't have much to add here.

The second is also fairly well understood, IMO, and the easiest path to delivering value to users. There we can add a lot of value quickly, without reinventing the entire CLI experience, or relying on third parties to implement new things we're providing support for. Broadly, the things that are higher on the backlog tend to fit into this category, and these are the ones that I believe my PMs would prefer I keep under wraps. I'll probably have some specs out early this year to help articulate some of these thoughts.

The last category was the one that was the least understood and the most open ended (and seemed like the threads you were most interested in 😉). It was also perhaps the most revolutionary. There was a lot of different ideas here on different ways for CLIs to provide richer TUIs. Some of that was things the console subsystem would need to support, while other seemed more like things that should be provided by a modern TUI library (like Terminal.Gui or Textualize). The closest concrete thing to come of that is some of the semantic markup stuff that Carlos is working on for a11y's sake. That's more of a start in the direction of defining richer regions of the output. IMO that would be best paired with a successor to ncurses, but that's not something we here on the Terminal team are gonna end up writing ourselves. This is the category of stuff that would really require a broad effort to coordinate with 3p devs and figure out exactly what needs to be built, rather than just doing it Field of Dreams style.

This is a real non-answer and I'm aware of that. A lot of this was already on the backlog in one shape or another, so this mostly just helped bump some things around. We've got a lot of other fundamentals to get out of the way before getting to some of these. I'd say stuff like that is really our priorities right now - tear out, better unicode, improved a11y support, stuff like that.

well, I wasn't expecting an answer at all, so thank you for what you did provide, even if it is vague and non-committal.

1. I think some of the things you're expecting here are the responsibility of the operating system, so there's not much the terminal can do about that. But you can already do things like setting the code page and querying the screen size with standard VT sequences.

2. A lot of this is covered by the DA1 report, which gives you the conformance level that the terminal supports, along with any extensions that is can handle beyond that base level. Supported modes can be queried with DECRQM, supported renditions with DECRQSS, and supported character sets with DECCIR. These queries may not be as simple as you would like, but they are at least part of an existing standard that has been around for decades.

3. This will depend on what modes you've selected, so I guess you can query the active modes to determine what keyboard and mouse reports you should be receiving, at least to some extent.

4. This could theoretically be achieved with S8C1T, but I don't know of anyone that supports it correctly for UTF-8 input. It's on my TODO list to implement in Windows Terminal, but there's not much point if nobody else supports it.

5. From past experience, I'd wager there is no chance of something like this being agreed upon. But it's probably safe to assume that most terminals support a subset of the VT100 sequences.

6. This is possible with synchronized output mode ?2026. It's not an official standard, but there are a fair number of terminals that support it, although not yet Windows Terminal.

7. This can be achieved with DECLRMM (left right margin mode) and DECSRLM (set left right margins).

8. If I've understood you correctly, this sort of thing can be done with VT pages, which are similar to the alternate screen buffer but much more powerful. You can choose to write to one page while another page is focused, and you can also copy content between pages. This is not yet supported in Windows Terminal, but it's on my TODO list.

9. I think there are VT sequences for something similar to this, but I don't know of any modern terminals that support them. Allowing an app to override system keys might also be considered a security risk.

10. I'm not sure I understand what you're suggesting here, but I think this is possibly outside the scope of the terminal.

@wcydo I deleted your previous comment because it was excessively rude, and I will be doing the same to this comment. Please refer to our code of conduct before you continue to engage in discussions here.

Of course you did. It simply is mind boggling that you're engaging in all this while you're leaving behind the most important factor and clearly the most obvious drawbacks of using this thing on daily basis. It is clearly slow and under-performing. I toned down my response by writing another comment simply listing my "wild" ideas for the terminal by stating that it must perform faster. It must open up instantly. It must not take 5 seconds to decide if an incorrect command is given. It must not take 3 seconds to fetch user path. I believe these are fair and true major drawbacks with this project.

You're yelling at the wrong people here. The terminal emulator doesn't control how long it takes to display your prompt or figure out what path you're running from, nor does it control how long it takes to reply telling you that a command wasn't found.

WT starts up in ~250ms from launch to interactivity, which is as far as we can tell near the "speed of light" for the frameworks we're using. If that is insufficient for you, you are welcome to use any other terminal emulator and go there to yell at them about how slowly your chosen shell starts up or processes your input.

To be entirely fair, I agree with some of the points you're making. Sure, PowerShell isn't the fastest shell on earth. That bothers me pretty much every day! The team who owns it and maintains their GitHub repository has also heard this feedback before and they certainly do care about it. After all, they live in PowerShell just as much as I do. They're impacted by it just as much as we are.

Practically speaking, this thread is not the appropriate place to vent your concerns. We--and our community--are allowed to discuss what cool things terminals could do in the years beyond 2023. It's not helpful and it's not on topic for you to show up in this specific thread and yell at us about things that the applications of today aren't doing up to your standards.

May I further refer you to What's the difference between a shell and a terminal?.

I hope that helps clarify!

Something like #5916 maybe?

I just read #5916. Indeed, it seems similar to the feature request you mentioned. In fact, the feature proposed in #5916 appears to be more abstract and flexible than my request.

While I can't fully imagine how the feature in #5916 would be implemented, it would presumably be more convenient if it's possible to toggle triggers on and off at will. For example, enabling them only when needed during work via the command palette, or activating them when opening a new tab with a specific profile. Additionally, it would be beneficial if the GUI could visually indicate whether a trigger is currently active.

Furthermore, it would be greatly appreciated if there was an option for each trigger to either continuously respond until manually disabled, or to become inactive after completing a series of responses.

Here's an example of what the JSON configuration might look like:

             "experimental.triggers": [
                 {
                     "type": "matchRegex",
                     "match": "Terminate batch job \\(Y/N\\)",
                     "command": { "action": "sendInput", "input": "y\n" },
+                    "persistent": true
                 }
             ]

i said stupid thing that conhost supports mouse tracking, seem it doesnt. i was testing ConEmu silently injected intance.. still DECRQM will be very useful

Conhost has supported mouse tracking for years now! It should support SGR, Xterm and Xterm UTF-8 encodings for Any, Button or (the third common kind that I cannot recall) tracking. vttest will attest to this.

In general, the purpose of this thread is to imagine what terminal emulation should be capable of in the future.

If you are having problems with things that should work today, or your imagination only extends to things that already exist, please file actual bugs so we can track them and prioritize them 😄

hmm, oke, i was starting my app from cmd.exe and powershell.exe and thought they pass mouse events, seem not. DECRQM will be a brand new shining in chrome feature for the future :]

another new feature:

resize event, like private mode number 1008 (for example) and the sequence <CSI> columns ; rows M

clarification of the wheel event:

how to extract wheel event from the input? how can terminal user distinguish vertical up from down and horizontal left from right. those arent in the docs anywhere. i found it in the ConEmu source code, me, terminal user who only should read docs, not terminal source code