On Fri, Feb 25, 2011 at 08:47:13PM +0100, Vegard Øye wrote:

I would love to work on a common project. Getting all our users on a single "platform" will likely give us more user extensions, which is my main motivation for doing what I do -- the great thing about sharing code is that others share back, too. :)

agree

How do you imagine we set up a common project? There is a couple of technical issues to sort out:

* Version control (Git vs. Mercurial)

The last time I checked (when I made the switch from SVN to Git), this list was rather Git-partial. (I actually argued for Mercurial then.) I have since grown to love how Git does a lot of things, like branching. Also, I have no experience with Mercurial.

I have no experience with git but I don't really care about the concrete VCS. I use mercurial, well, because I've used it for several years know and at that time git's documentation was poor. But IMO it is just a tool to get a job done, both are almost equivalent in all important aspects and both provide all the things we need. Most of the time no one really needs the fancy stuff both provide but only log/push/commit/merge and branch. And I can use git-reps with mercurial anyway (and probably the other way around) ;)

How about you?

* Hosting

If we go with Git, I suggest Gitorious as a hosting provider: it is project-oriented rather than person-oriented. I know of no project-oriented Mercurial providers.

Neither I do (perhaps sourceforge? There could be some non-VCS-specific hosting providers). Gitorious should be okay.

* Merging

Where do we start? What parts will benefit from a rewrite, and what parts can be included as-is?

The most important part is obviously the core, which switches between states and connects motions and commands. This core should be as small as possible but flexible. The following things have to be considered before we start because they have significant influence on the rest: - repeating. Vim-mode uses keyboard-macro and those have to be recorded. Recording them can be tricky if one keyboard macro calls another one. Furthermore insert-mode is different because it contains of several independent key-sequences. Using key-sequences has some benefits but also some downsides, e.g., when lazy completion packages are used - undo, vim usually undos everything done in insert-mode at once while emacs undos insertion in steps. - catching of commands and motions. In each state the execution of commands and motions has different effects and therefore some special function has to be executed in each mode to deal with commands and motions. In general I see two possibilities how this special function can be executed. Either one hooks into pre- and/or post-command hooks and checks whether the command/motion about to be executed is a Vim operator and in this case do whatever is required according to the current state. Or each command and motion is defined in a special way in order to call the state specific handler when it is executed. vim-mode uses the latter one because I try to avoid pre- and post-command hooks if possible. - Together with the point above the definition of commands and motions has to be clarified. This is the most important part because almost all user-extensions will be commands and motions and not special modes or other funny things. - What to do with non-vim commands and motions? There should be a simple rule so they work with vim in most cases as expected, but if no meta-information is provided a non-vim command may always be problematic (e.g., in operator-pending mode). - What to do with commands that change the current buffer? This is problematic if some buffer-local variables are involved. - How to switch states and how to represent states (usually as minor-modes, perhaps with some further meta-data like cursor, info-message, ...) - What is a motion? There are a few fundamental differences between buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs knows just the offset. The choice how to represent coordinates can be crucial. And of course there is that funny newline character, which is usually invisible in Vim except for empty lines. Because a motion is a important concept for many parts including operators and visual-mode. Probably a motion is represented by some abstract data type and this data-type is an important part of the interface a user must understand in order to write new motions and commands. Therefore it should be well-defined from the beginning because it cannot be changed easily afterwards. These things should make the core and this has to be done right. Once the core is ready most motions and commands are quite easy, probably most can be copied from vim-mode or vimpulse, especially scrolling and window commands, but also many others. If the definition of motions and commands is clear, the implementation of those should be almost independent from the core and I think this is very important. A single command should just do what its purpose is and not interfere with the rest. In fact, there are only two (or three) difficult commands: yank, delete and paste, but I hope the implementation in vim-mode is quite good know (it uses yank-handlers for all three kinds of yanking and also provides "yank-pop" stuff). Note that these commands are (almost?) independent from the rest of vim-mode, too. Some parts of vim-mode are almost independent (or "orthogonal", although I always have to smile when I read this word ;)) of the rest, among them ex-mode, interactive search and the visual-stuff of visual-mode. This should remain true for the new project.

Whatever we decide, I think extensibility should be a prime goal.

I agree, whereas I would put the focus on new commands and motions. The rest should be possible by defining clear interfaces. What could be useful are frameworks or perhaps only some functions which help defining more involved motions like text-objects. We should provide helper-functions for typical text-objects like parentheses, blocks, whatever. vim-mode (and vimpulse, too) contains a few of those functions which are again independent.

In that regard, neither Vimpulse's nor vim-mode's keymap handling are quite as flexible as I'd like (I want to assign vi/Insert/Visual bindings to Emacs minor and major modes; this was a lot of work to accomplish under Viper).

I read your comments and they sound reasonable. Adding auxiliary keymaps should not be that difficult. As I remember correctly, local keymaps were the biggest problem in vim-mode, but major and minor-mode specific should be relatively easy. Of course this is very important and should be done as one of the first things. Note that vim-mode arranges keymaps in another hierarchy, too, which allows to enable only few keybindings in certain major modes, e.g., only movement and window-commands in info-mode (somehow like viper-on-more-modes I think). The difference is that many standard-keybindings are not available in this case, e.g., insert-mode.

My suggestion is to write an extensible core from scratch, and then "plug" the more mature parts into that framework.

Agree again. But we should try to cover as much aspects of the core as possible from the beginning. Otherwise we will need to patch the core later on as other functions are required and this can be problematic. I hope that our experiences with vimpulse and vim-mode can help us.

For what it's worth, I've pushed a sparse starting point to the "development" branch. The comments explain the keymap issue in more detail.

http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-states.el http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-tests.el

* Unit tests

I would really like to have a modicum of test coverage this time around. I don't how many Visual mode regressions I could have prevented if I had started writing tests earlier, but from my experience, some of the issues pertaining to Emacs' region (among other things) are naturally hairy.

Regarding test frameworks, I hear Christian Ohler's ert.el is now included with Emacs trunk, so we could go with that:

http://www.emacswiki.org/emacs/ErtTestLibrary

I never used any of those libraries but we should try. The main problem is to cover all those funny special cases that arise (what happens at the end of the buffer, on empty lines, ...) - most bugs found in vim-mode are of this kind. This is another point which is currently very bad in vim-mode. How to deal with errors, should there be a unique signalling scheme, where to handle them, ... Btw, Emacs 23 contains visual-line-mode which is quite useful. But most commands work on buffer lines not on screen lines. Should we try to support visual-lines, too (optionally)? If yes this has to be considered for motions and commands, especially for operator-stuff, too, and we should think of it from the beginning, because many commands have to be implemented differently for visual-lines. Another thing somehow related to tests is documentation. We should not only write developer-documentation but also user-documentation right from the start. This should contain instructions for all important concepts and also examples for new motions and commands of all kind. Where should we put this documentation? EmacsWiki? Hosting site? Perhaps as tex-info?

* Name?

Something new? Something catchy? Suggestions?

Hm, I think a new cool name would be good. I prefer short names or at least names that allow short prefixes for function names ;) I have no good idea so far, but I'll think about it ... I really hope we can start a common project. Frank

On 2011-02-25 22:20 +0100, Frank Fischer wrote:

Gitorious should be okay.

Then Gitorious it is! :)

- repeating. Vim-mode uses keyboard-macro and those have to be recorded. Recording them can be tricky if one keyboard macro calls another one. Furthermore insert-mode is different because it contains of several independent key-sequences. Using key-sequences has some benefits but also some downsides, e.g., when lazy completion packages are used

A big benefit of recording the keys is that it removes the need for repeat code in the commands themselves. For example, I have written some custom insertion commands for dealing with Lisp code (e.g., "Insert after S-exp"); if I want them to repeat correctly under Viper, I have to edit the repeat history from within the commands, which is /extremely/ cumbersome. By contrast, a repeat system which monitors keystrokes and the current state would simply take note that I have entered Insert state, and update the repeat history accordingly. As you point out, some keystrokes are troublesome: "M-/", which calls `dabbrev-expand', may expand to different words in different places in the buffer. One solution may be to define the repeat history as a collection of keystrokes and buffer insertions, e.g., ([a b c] "inserted text" [d e]) Most commands are remembered as keystrokes (vectors), but certain ones, like `dabbrev-expand', are remembered as insertions (strings) instead. The repeat system would use a "blacklist" of unsafe commands like `dabbrev-expand'; if the current command is listed, the system creates a marker at point (in `pre-command-hook'), lets the command execute, and then (in `post-command-hook') extracts the text between point and the marker, which goes into the repeat history like above.

- undo, vim usually undos everything done in insert-mode at once while emacs undos insertion in steps.

This is not that difficult in itself, but becomes more complex when loading a custom undo package like undo-tree.el. (I would really like to use undo-tree.el in place of redo.el; it features a graphical interface which totally outclasses Vim's :undolist.) There is a case for making Vim-like "bulk undoing" toggleable: Emacs' fine-grained undoing can be valuable in some circumstances. The default should be Vim-like behavior, of course.

- catching of commands and motions. In each state the execution of commands and motions has different effects and therefore some special function has to be executed in each mode to deal with commands and motions. In general I see two possibilities how this special function can be executed. Either one hooks into pre- and/or post-command hooks and checks whether the command/motion about to be executed is a Vim operator and in this case do whatever is required according to the current state. Or each command and motion is defined in a special way in order to call the state specific handler when it is executed. vim-mode uses the latter one because I try to avoid pre- and post-command hooks if possible.

Ah, the relationship between operators and motions. This is an area where Vimpulse and vim-mode differ somewhat. In vim-mode, as I understand it, motions are passed to operators by parameter (or the operator reads a motion from the keyboard when called interactively). The operator executes the received motion to determine what text to act on. The resulting pair of buffer positions is used internally. In Vimpulse, the buffer positions /are/ the parameters (BEG and END). Operators work like region commands (i.e., (interactive "r")) in that they either receive the positions from the caller, or figure them out for themselves when called interactively. (Vimpulse also possesses some facilities for "passing a motion to an operator", which executes the specified motion and feeds its range to the operator. This is only used by the repeat system, though, and it would be totally superfluous if a keystroke-based system was in place.) For the sake of reusability, I recommend defining operator commands in the same way as region commands: with two parameters, BEG and END, and an `interactive' specification for determining their values by reading a motion from the keyboard.

- What to do with non-vim commands and motions? There should be a simple rule so they work with vim in most cases as expected, but if no meta-information is provided a non-vim command may always be problematic (e.g., in operator-pending mode).

Well, we will likely have one macro for defining operators, and another for defining motions. Both can be set up to index all defined operators/motions for later lookup, which will give us some information to go on. By the way, I also think that all of Emacs' standard movement commands should be regarded as motions. We thus get a large list of commands which move the cursor, and can restrict Operator-Pending state to those commands. Alternatively, we can maintain a "blacklist" of Emacs commands which should /not/ be executed. In any case, I don't think it's overly problematic if a nonsensical command should get called; any damage will be undoable, after all. It's probably better to err on the side of allowing too many commands than too few.

- What to do with commands that change the current buffer? This is problematic if some buffer-local variables are involved.

Right, which means that the repeat system cannot be buffer-local. I think it would just be confusing if it was, anyway.

- How to switch states and how to represent states (usually as minor-modes, perhaps with some further meta-data like cursor, info-message, ...)

We'll have a macro for defining those too, of course. :) I suggest that each state's toggle function be a command; e.g., in Vimpulse's development code, vi state defines the command `vimpulse-vi-state' for entering that state, emacs state defines `vimpulse-emacs-state', and so on. The command also has an optional argument which, if negative, disables the state; this is used internally when switching from one state to another.

- What is a motion? There are a few fundamental differences between buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs knows just the offset. The choice how to represent coordinates can be crucial. And of course there is that funny newline character, which is usually invisible in Vim except for empty lines. Because a motion is a important concept for many parts including operators and visual-mode.

I think all of Emacs' regular movement commands should be valid motions. Furthermore, all of Emacs' selection commands should be valid text objects. In Vimpulse, "y C-x h" and "C-x h y" are equally valid ways of copying the whole buffer, and "d M-e" works just as well for deleting a sentence as does "d)". Given the above, the only thing which separates Emacs' movement commands from the other motions is their lack of a /type/. For example, the motion "j" has a type of `line', which means that "dj" deletes two whole lines. What a type is, then, is a way to transform two buffer positions to two other positions -- in this case, to two whole lines.

Probably a motion is represented by some abstract data type and this data-type is an important part of the interface a user must understand in order to write new motions and commands. Therefore it should be well-defined from the beginning because it cannot be changed easily afterwards.

I don't think it needs to be very abstract. I would rather try to keep things from getting too abstract, actually. One of the design rules I laid down in Vimpulse's new "TODO" list was that it should require as few packages as possible, and not invent new stuff unless it's absolutely necessary. The way I understand motions, they are just regular movement command plus a type. The type can be stored as a symbol property, e.g., (put 'motion 'type 'line). The type itself, of course, must also be defined; for this, a minimal system may be warranted, since both Operator-Pending state and Visual state use types. I've pushed a currently unused "type system" to "development": http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-types.el Now, it's a bit more involved than it has to be; the routines for reselecting (e.g., two lines anywhere in the buffer) are unnecessary given a keystroke-based repeat system. The advantages of a systematic approach is that it can be reused by Operator-Pending state and Visual state, making the overall code simpler. The only heavy lifting Visual state ends up doing (once initialized) is highlight the different types, which is rather trivial (with the exception of `block' ...).

In fact, there are only two (or three) difficult commands: yank, delete and paste, but I hope the implementation in vim-mode is quite good know (it uses yank-handlers for all three kinds of yanking and also provides "yank-pop" stuff).

Yes, this is exactly the right way to do it. I think the actual yank handlers should be specified by the type system, so that `line' has one handler, `block' has another, and so on. That way, we avoid hard-coding these associations into the "yank" command.

What could be useful are frameworks or perhaps only some functions which help defining more involved motions like text-objects. We should provide helper-functions for typical text-objects like parentheses, blocks, whatever. vim-mode (and vimpulse, too) contains a few of those functions which are again independent.

Agreed; we should have a macro for defining text objects, too. To recap, we now have the following define macros: * `define-state' * `define-type' * `define-motion' * `define-text-object' * `define-operator' Do we need more than these?

...

http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-states.el

I read your comments and they sound reasonable. Adding auxiliary keymaps should not be that difficult. As I remember correctly, local keymaps were the biggest problem in vim-mode, but major and minor-mode specific should be relatively easy. Of course this is very important and should be done as one of the first things.

Actually, the current code already implements auxiliary keymaps, although I've not tested it thoroughly.

Note that vim-mode arranges keymaps in another hierarchy, too, which allows to enable only few keybindings in certain major modes, e.g., only movement and window-commands in info-mode (somehow like viper-on-more-modes I think). The difference is that many standard-keybindings are not available in this case, e.g., insert-mode.

Yes, this is very sensible. Motions and text objects can be bound in their own keymap (or state), and that keymap is inherited by vi state, Visual state, and so on. Modes which do not work well in vi state, can come up in the motion state instead. Any shortcomings can be alleviated on a per-mode basis with mode-specific state bindings.

...

Regarding test frameworks, I hear Christian Ohler's ert.el is now included with Emacs trunk, so we could go with that:

I never used any of those libraries but we should try. The main problem is to cover all those funny special cases that arise (what happens at the end of the buffer, on empty lines, ...) - most bugs found in vim-mode are of this kind.

This can be tested by executing a keyboard macro in a temporary buffer. A lot of Vimpulse's early tests do this, e.g., (deftest test-change-undo "Change a word and undo." (execute-kbd-macro "wcwfoo^[u") (assert-string= (buffer-substring 1 51) ";; This buffer is for notes you don't want to save"))

This is another point which is currently very bad in vim-mode. How to deal with errors, should there be a unique signalling scheme, where to handle them, ...

In "indirect" cases, like when executing a motion to get a text range, I think any errors should be suppressed. When executing the motions by themselves, however, we should signal any problems with `error'.

Btw, Emacs 23 contains visual-line-mode which is quite useful. But most commands work on buffer lines not on screen lines. Should we try to support visual-lines, too (optionally)? If yes this has to be considered for motions and commands, especially for operator-stuff, too, and we should think of it from the beginning, because many commands have to be implemented differently for visual-lines.

Yes, we should definitely support screen lines; I have "j" and "k" bound to `next-line' and `previous-line' because they work better in visual-line-mode. In Vim, one uses "gj" and "gk" to move by screen lines. I think this is very inefficient and would prefer an option for the regular commands. It may of course be disabled by default.

Another thing somehow related to tests is documentation. We should not only write developer-documentation but also user-documentation right from the start. This should contain instructions for all important concepts and also examples for new motions and commands of all kind. Where should we put this documentation? EmacsWiki? Hosting site? Perhaps as tex-info?

Texinfo, definitely. I've had my share of arbitrary wiki syntax and never-ending comment blocks. A text file under version control is wiki-like enough for me.

Hm, I think a new cool name would be good. I prefer short names or at least names that allow short prefixes for function names ;)

For the sake of brainstorming, I've grepped through /usr/share/dict for anything containing "vi". There isn't much, unfortunately, so I'll start with some acronyms comprising three letters: * avi, evi (augmented/extensible vi) * via, vie (vi augmented/extended) * vil (vi layer) * xvi (vi plus a cool letter) * yvi (yet a(nother) vi) Things get slightly more descriptive when we allow four letters: * evil (extensible vi layer; this almost works too well) * nuvi (new (unlimited) vi) * vibe (vi is beautiful?) * vici (veni, vidi ...) * vimu (vi(m) emulation; vim unlimited; vi made usable) * vini (vi new and improved) * viva (viva la vi!) * yavi (yet another vi) With five letters or more, there are many proper names to choose from (elvis is the name of a vi clone): alvin, levi, david, vidal, vince. And we have: * anvil (a new vi layer; doesn't sound very light, though) * vibrant (the soothing, relaxing, vibrating editor) * vigil (vi greatly improved layer-something) * vital (vi is vital to, say, avoiding RSI ...) More letters would be inconvenient; I want the name and code prefix to be one and the same. (I use an abbreviated prefix in viper-in-more-modes.el, and I hate the sight of it.) Of the above, I rather like "vimu". It's short, it's sweet, and it can be read as both "vi emulation" and "Vim emulation". Although I think Vim sets the baseline for what a modal editor should support, it's not the end of the road for me. A sufficiently extensible implementation will enable the user to explore uncharted territory. -- Vegard

Am Samstag, 26. Februar 2011 schrieb Vegard Øye:

On 2011-02-25 22:20 +0100, Frank Fischer wrote:

...

Gitorious should be okay.

Then Gitorious it is! :)

yuk ;)

A big benefit of recording the keys is that it removes the need for repeat code in the commands themselves. For example, I have written some custom insertion commands for dealing with Lisp code (e.g., "Insert after S-exp"); if I want them to repeat correctly under Viper, I have to edit the repeat history from within the commands, which is /extremely/ cumbersome. By contrast, a repeat system which monitors keystrokes and the current state would simply take note that I have entered Insert state, and update the repeat history accordingly.

As you point out, some keystrokes are troublesome: "M-/", which calls `dabbrev-expand', may expand to different words in different places in the buffer. One solution may be to define the repeat history as a collection of keystrokes and buffer insertions, e.g.,

([a b c] "inserted text" [d e])

Most commands are remembered as keystrokes (vectors), but certain ones, like `dabbrev-expand', are remembered as insertions (strings) instead. The repeat system would use a "blacklist" of unsafe commands like `dabbrev-expand'; if the current command is listed, the system creates a marker at point (in `pre-command-hook'), lets the command execute, and then (in `post-command-hook') extracts the text between point and the marker, which goes into the repeat history like above.

I wonder if it is that easy. What kind of repeation is intended? Sometimes you want to repeat the inserted text, but another person wants to repeat the expansion context sensitive. I really have to think about a good compromise (of course dabbrev-expand may yield different results - but are they intendend? does this happend regularly or is it just a rare case?). Some further comments. First using markers does not work in all cases, only if point is placed right after the inserted text. Other insertion packages may place point, e.g., somewhere in middle of the inserted text. Propably it's better to use after-change-functions. Second in general I do not like blacklists. The reason is that there are many Emacs packages out there each may define its own commands. Furthermore each new Emacs version could introduce new commands and we have to keep the blacklist up-to-date. Of course, each user can modify this list and adjust it to the packages she uses. But I think this may be cumbersome for the average user. And sometimes it may be quite difficult to find out which key-sequence to drop - I thing of auto-completion packages where during completion the user executes several commands and only few of them (the final RET?) really inserts some text, and this completions are often triggered by a timer, not by an explicit command. I would prefer some good default rule which works in most cases (blacklists are okay for rare cases). This does not mean I have a good solution for these problems just a few things to consider (and I often run into the auto-completion problem in programming, so this is not a rare case).

...

- undo, vim usually undos everything done in insert-mode at once while emacs undos insertion in steps.

This is not that difficult in itself, but becomes more complex when loading a custom undo package like undo-tree.el. (I would really like to use undo-tree.el in place of redo.el; it features a graphical interface which totally outclasses Vim's :undolist.)

I never used :undolist so I can't say anything useful to it. For me redo has always been okay, so this is up to you ;)

There is a case for making Vim-like "bulk undoing" toggleable: Emacs' fine-grained undoing can be valuable in some circumstances. The default should be Vim-like behavior, of course.

...

- catching of commands and motions. In each state the execution of commands and motions has different effects and therefore some special function has to be executed in each mode to deal with commands and motions. In general I see two possibilities how this special function can be executed. Either one hooks into pre- and/or post-command hooks and checks whether the command/motion about to be executed is a Vim operator and in this case do whatever is required according to the current state. Or each command and motion is defined in a special way in order to call the state specific handler when it is executed. vim-mode uses the latter one because I try to avoid pre- and post-command hooks if possible.

Ah, the relationship between operators and motions. This is an area where Vimpulse and vim-mode differ somewhat. In vim-mode, as I understand it, motions are passed to operators by parameter (or the operator reads a motion from the keyboard when called interactively). The operator executes the received motion to determine what text to act on. The resulting pair of buffer positions is used internally.

That's wrong. Vim-mode almost behaves like what described below. Perhaps it's time to tell you how the things work in vim-mode (and I think this is not too bad, but certainly can be improved). Motions in vim-mode are represented by the vim:motion struct. This struct contains the two buffer positions along with the intended motion type (inclusive, exclusive, linewise, block). It is this data-structure that is passed to operators. So the single motion argument for operators in vim-mode is rather equivalent to the two parameters BEG END in vimpulse plus the type of the motion. Usually operators use the motion type to behave differently (but this is not required). If we had CLOS-like methods one would propably dispatch on the motion type. A motion in vim-mode is function that returns a vim:motion structure. This can be done either implicit or explicit. Explicit means that the function body really creates the structure and returns it. This is usually the case for text-objects (they are just usual motions in vim-mode) or some special commands like , or ; which return motions of different types depending on the previous f F t T command. Implicit means that the function does not return an explicit vim:motion structure. In this case it is enough to move point. Then the magic within vim:defmotion ensures that a vim:motion structure is automatically created depending on the default type of the motion and the movement of point. Implicit is therefore enough for almost all simple motions what makes them one-liners. Note that from the caller's point of view it makes no different to call an implicit or explicit returning motion - the caller always receives the vim:motion structure. When executing an operator the operator is not executed immediately. Instead the execution of its body is postponed until a motion has been selected. When the motion has been selected, the handler of operator-pending-mode first executes the motion to create the vim:motion structure and then passes this structure to the operator which is executed afterwards. In visual-mode the vim:motion structure is not created by a motion but w.r.t. to the current visual selection. All in all, the representation of a motion is very simple, just three values, BEG END and TYPE. I decided to use keyword arguments because in vim-mode commands and motions may take a different set of arguments. E.g., a command may take - count - motion - register - additional argument (char, text, file, buffer, ...) whereas the latter are only usefull for ex-commands - force (only for ex) Also note that in vim-mode each command can be bound in ex-mode AND normal-mode/visual-mode. In my philosophy ex-mode is just another mode with own keybindings, although they are read in a different way.

In Vimpulse, the buffer positions /are/ the parameters (BEG and END). Operators work like region commands (i.e., (interactive "r")) in that they either receive the positions from the caller, or figure them out for themselves when called interactively. (Vimpulse also possesses some facilities for "passing a motion to an operator", which executes the specified motion and feeds its range to the operator. This is only used by the repeat system, though, and it would be totally superfluous if a keystroke-based system was in place.)

I really like the idea of making operators as close to usual emacs commands as possible. This is certainly possible for character motions, but how do you tell which kind of motion the region belongs to? Call different commands depending on the type? Modify point and mark? The latter is possible for character and linewise motions, but what about block? Furthermore modifying point and mark may be problematic. Several commands have rules where to place point after execution depending on the original position of point (before modification to respect the visual region). I can think of defining operators with arguments (BEG END &optional (type 'char) register) and an apropriate interactive specification. This would enable them to behave like normal region-operations but gives the additional possibilities to pass further important information. But then there is the problem how to pass the additional arguments interactively. But it sounds as if you have a good idea how to do this ;)

For the sake of reusability, I recommend defining operator commands in the same way as region commands: with two parameters, BEG and END, and an `interactive' specification for determining their values by reading a motion from the keyboard.

...

- What to do with non-vim commands and motions? There should be a simple rule so they work with vim in most cases as expected, but if no meta-information is provided a non-vim command may always be problematic (e.g., in operator-pending mode).

Well, we will likely have one macro for defining operators, and another for defining motions. Both can be set up to index all defined operators/motions for later lookup, which will give us some information to go on. By the way, I also think that all of Emacs' standard movement commands should be regarded as motions.

As above, this would require a blacklist and have to keep the list up-to-date. And other packages may introduce new motions. So in all cases we need very good defaults so noone has to modify that list in almost all cases.

We thus get a large list of commands which move the cursor, and can restrict Operator-Pending state to those commands. Alternatively, we can maintain a "blacklist" of Emacs commands which should /not/ be executed. In any case, I don't think it's overly problematic if a nonsensical command should get called; any damage will be undoable, after all. It's probably better to err on the side of allowing too many commands than too few.

...

- What to do with commands that change the current buffer? This is problematic if some buffer-local variables are involved.

Right, which means that the repeat system cannot be buffer-local. I think it would just be confusing if it was, anyway.

Of course it is not, but there may be other buffer-local variables. In vim-mode a tackle this problem by saving the new-buffer when a command returns, switching back to the original buffer, do the necessary work and switch to new buffer at the end. This seems to work quite well but can certainly be improved.

...

- How to switch states and how to represent states (usually as minor-modes, perhaps with some further meta-data like cursor, info-message, ...)

We'll have a macro for defining those too, of course. :)

I suggest that each state's toggle function be a command; e.g., in Vimpulse's development code, vi state defines the command `vimpulse-vi-state' for entering that state, emacs state defines `vimpulse-emacs-state', and so on. The command also has an optional argument which, if negative, disables the state; this is used internally when switching from one state to another.

Should we use standard minor-modes for this? I do this in vim-mode and it seems to be okay, but I wonder if there are any downsides.

...

- What is a motion? There are a few fundamental differences between buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs knows just the offset. The choice how to represent coordinates can be crucial. And of course there is that funny newline character, which is usually invisible in Vim except for empty lines. Because a motion is a important concept for many parts including operators and visual-mode.

I think all of Emacs' regular movement commands should be valid motions. Furthermore, all of Emacs' selection commands should be valid text objects. In Vimpulse, "y C-x h" and "C-x h y" are equally valid ways of copying the whole buffer, and "d M-e" works just as well for deleting a sentence as does "d)".

Hm I think d M-e is okay besides the fact that a sentence is slightly different in Emacs and Vim (I think), but never mind. The first example seems to be very special case. But if I understand correctly you just want a function which changes point and mark and toggle transient-mark-mode to behave like a text-object ... or to enable visual-mode. Sounds reasonable to me although I'm always afraid of compatibility issue between Emacs regions and visual-regions. Is it easy to detect that a certain command behaves like this or is it another blacklist variant? ;)

Given the above, the only thing which separates Emacs' movement commands from the other motions is their lack of a /type/. For example, the motion "j" has a type of `line', which means that "dj" deletes two whole lines. What a type is, then, is a way to transform two buffer positions to two other positions -- in this case, to two whole lines.

...

Probably a motion is represented by some abstract data type and this data-type is an important part of the interface a user must understand in order to write new motions and commands. Therefore it should be well-defined from the beginning because it cannot be changed easily afterwards.

I don't think it needs to be very abstract. I would rather try to keep things from getting too abstract, actually. One of the design rules I laid down in Vimpulse's new "TODO" list was that it should require as few packages as possible, and not invent new stuff unless it's absolutely necessary.

I think we both have the same feeling of it (see above). In fact, vim:motion in vim-mode comes with a couple of functions which do exactly this (returning appropriate buffer positions w.r.t. the motion type).

The way I understand motions, they are just regular movement command plus a type. The type can be stored as a symbol property, e.g., (put 'motion 'type 'line). The type itself, of course, must also be defined; for this, a minimal system may be warranted, since both Operator-Pending state and Visual state use types. I've pushed a currently unused "type system" to "development":

http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-typ es.el

Now, it's a bit more involved than it has to be; the routines for reselecting (e.g., two lines anywhere in the buffer) are unnecessary given a keystroke-based repeat system. The advantages of a systematic approach is that it can be reused by Operator-Pending state and Visual state, making the overall code simpler. The only heavy lifting Visual state ends up doing (once initialized) is highlight the different types, which is rather trivial (with the exception of `block' ...).

...

In fact, there are only two (or three) difficult commands: yank, delete and paste, but I hope the implementation in vim-mode is quite good know (it uses yank-handlers for all three kinds of yanking and also provides "yank-pop" stuff).

Yes, this is exactly the right way to do it. I think the actual yank handlers should be specified by the type system, so that `line' has one handler, `block' has another, and so on. That way, we avoid hard-coding these associations into the "yank" command.

At this point I completely disaggree. I don't like the idea of coding behaviours of yank/delete and so on into the type system for two reasons. First of all, yank/delete should not be different from any other command. A special handling of them in the type system seems to be confusing to me. The other reason is that there may be even more commands that have different behaviour depending on the motion type, not just yank/delete, even commands that we cannot think of because we want *extensibility*. I prefer a type to be as simple as possible and to have as less additional functions and responsibilities as possible. The type should deal with its buffer positions. Not more. Anything else like handling of the visual-region or commands like yank/delete do not belong here. I think the main difference here is that you want to stick with the standard interface of Emacs for operators (only BEG END) while I want to provide additional information (especially motion type). So in my world it would be the responsibility of the command to behave differently on different motion types. Of course then your commands need to be sensitive to the motion types - but you have to deal with this sensitivity anyway, nothing prevents you from that. The question is where. Of course it would be nice if operators also behave like usual Emacs commands when used in such a context. This was the reason of my suggestion above (BEG END &optional type ...). In fact, what I would need is a dispatch on the motion type, i.e., (yank beg end 'char) calls another function than (yank beg end 'line). I do not know of any such functionality in Emacs so propably it ends up with manual dispatch (case type (line (yank-line ...)))) and so on. I do not think this is too bad and certainly not more complicate than you suggestion. But I think it provides a better separation of code. Note that it is okay for me to have certain functions with update visual region according to a motion type. But since visual-mode is the only place where this is required it should go there. Of course in a structured way.

...

What could be useful are frameworks or perhaps only some functions which help defining more involved motions like text-objects. We should provide helper-functions for typical text-objects like parentheses, blocks, whatever. vim-mode (and vimpulse, too) contains a few of those functions which are again independent.

Agreed; we should have a macro for defining text objects, too.

To recap, we now have the following define macros:

* `define-state' * `define-type' * `define-motion' * `define-text-object' * `define-operator'

Do we need more than these?

Well there are also other commands that are not operators. E.g. paste requires at least two arguments, the count and the register. Other commands require additional character arguments like "r" or motions like "f". Of course those commands can read that character themselves. But I prefer passing the character as an argument because then this command can also be used non-interactively (useful, e.g., for commands like , and ;).

...

...

http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse- states.el

I read your comments and they sound reasonable. Adding auxiliary keymaps should not be that difficult. As I remember correctly, local keymaps were the biggest problem in vim-mode, but major and minor-mode specific should be relatively easy. Of course this is very important and should be done as one of the first things.

Actually, the current code already implements auxiliary keymaps, although I've not tested it thoroughly.

...

Note that vim-mode arranges keymaps in another hierarchy, too, which allows to enable only few keybindings in certain major modes, e.g., only movement and window-commands in info-mode (somehow like viper-on-more-modes I think). The difference is that many standard-keybindings are not available in this case, e.g., insert-mode.

Yes, this is very sensible. Motions and text objects can be bound in their own keymap (or state), and that keymap is inherited by vi state, Visual state, and so on. Modes which do not work well in vi state, can come up in the motion state instead. Any shortcomings can be alleviated on a per-mode basis with mode-specific state bindings.

...

...

Regarding test frameworks, I hear Christian Ohler's ert.el is now included with Emacs trunk, so we could go with that:

I never used any of those libraries but we should try. The main problem is to cover all those funny special cases that arise (what happens at the end of the buffer, on empty lines, ...) - most bugs found in vim-mode are of this kind.

This can be tested by executing a keyboard macro in a temporary buffer. A lot of Vimpulse's early tests do this, e.g.,

(deftest test-change-undo "Change a word and undo." (execute-kbd-macro "wcwfoo^[u") (assert-string= (buffer-substring 1 51) ";; This buffer is for notes you don't want to save"))

Note that executing a keyboard macro makes testing certain aspects very difficult, namely those that behave differently when executed in a keyboard-macro or not. I think of the repeating system which must take care of keyboard-macros.

...

This is another point which is currently very bad in vim-mode. How to deal with errors, should there be a unique signalling scheme, where to handle them, ...

In "indirect" cases, like when executing a motion to get a text range, I think any errors should be suppressed. When executing the motions by themselves, however, we should signal any problems with `error'.

I think errors should never be suppressed. The operator should just be canceled but the error should be shown. It is always dangerous to suppress errors (why does this not work? There is no error, right?) Note that there are other error-symbols in use like end-of-buffer and the like. I think we should use them whenever possible. But should we use special error-symbols for errors that occur in our code?

...

Btw, Emacs 23 contains visual-line-mode which is quite useful. But most commands work on buffer lines not on screen lines. Should we try to support visual-lines, too (optionally)? If yes this has to be considered for motions and commands, especially for operator-stuff, too, and we should think of it from the beginning, because many commands have to be implemented differently for visual-lines.

Yes, we should definitely support screen lines; I have "j" and "k" bound to `next-line' and `previous-line' because they work better in visual-line-mode. In Vim, one uses "gj" and "gk" to move by screen lines. I think this is very inefficient and would prefer an option for the regular commands. It may of course be disabled by default.

Yes it is definitely a difference for commands to work on screen-lines or visual-lines. Note that deleting a visual line may have funny results because the other lines may change a little bit, too (long words deleted which caused line-breaks). Furthermore block-motions do not really make sense for visual lines. I would definitely not use visual-line movements as defaults but document very well how it can be configured, because dealing with visual lines (as a user) may be very inconsistend.

...

Another thing somehow related to tests is documentation. We should not only write developer-documentation but also user-documentation right from the start. This should contain instructions for all important concepts and also examples for new motions and commands of all kind. Where should we put this documentation? EmacsWiki? Hosting site? Perhaps as tex-info?

Texinfo, definitely. I've had my share of arbitrary wiki syntax and never-ending comment blocks. A text file under version control is wiki-like enough for me.

Very good, texinfo allows for nice inline documentation via :help ;)

...

Hm, I think a new cool name would be good. I prefer short names or at least names that allow short prefixes for function names ;)

For the sake of brainstorming, I've grepped through /usr/share/dict for anything containing "vi". There isn't much, unfortunately, so I'll start with some acronyms comprising three letters:

* avi, evi (augmented/extensible vi) * via, vie (vi augmented/extended) * vil (vi layer) * xvi (vi plus a cool letter) * yvi (yet a(nother) vi)

Things get slightly more descriptive when we allow four letters:

* evil (extensible vi layer; this almost works too well) * nuvi (new (unlimited) vi) * vibe (vi is beautiful?) * vici (veni, vidi ...) * vimu (vi(m) emulation; vim unlimited; vi made usable) * vini (vi new and improved) * viva (viva la vi!) * yavi (yet another vi)

"evil" really gave me a smile ;) I thought of something like "VeM" but have no good idea for its interpretation. "vimu" sounds good, too. Bye, Frank

On 2011-02-28 14:09 +0100, Frank Fischer wrote:

The more I think about it I wonder if this is really the right idea using the full power of interactive. One could easily use custom interactive statements with lisp expressions like

(interactive (list (region-beginning) (region-end) (vimu:current-motion-type) (vimu:current-motion-register)))

(or similar) to provide all the required information. I think this would fit nicely into the usual Emacs commands and provides the additional information for vimu and they can be called non-interactively as any other function. Of course we can hide this rather complicate interactive statements behind some macros. I think this could solve all issues I mentioned in the last post and should be a good compromise.

Sounds good to me.

The other question that comes into my mind is the following. In vim-mode, as I told, I prevent an operator to be executed until the following motion has been executed. This can be done as above if the interactive form uses some kind of recursive edit (right?).

I don't understand ... the motion is executed in the operator's `interactive' specification, /before/ the operator's body. When you press "dw", "d" calls `delete', `delete' reads "w", "w" is bound to `forward-word', `forward-word' is executed to get buffer positions, and the positions are passed to the operator's arguments. Then the operator's body is executed. Are we talking about the same thing?

And I have a further question. We must certainly specify some additional properties for commands, e.g., some commands are repeatable while others are not repeatable, some should preserve visual-mode (e.g., several window and scrolling commands which are not really motions - they cannot be used with an operator and sometimes don't even change point) while others disable visual-mode (most editing commands). How do we specify theses properties?

I prefer to use keyword arguments: (vimpulse-define-operator vimpulse-indent (beg end) "Indent text." :repeat nil :whole-lines t (indent-region beg end nil)) These determine how the operator is defined. -- Vegard

On 2011-02-28 21:57 +0100, Frank Fischer wrote:

On 2011-02-28 21:45 +0100, Frank Fischer wrote:

...

On Mon, Feb 28, 2011 at 04:23:21PM +0100, Vegard Øye wrote:

...

If it is possible to reliably distinguish linear changes from non-linear ones in `after-change-functions', we can choose the appropriate representation accordingly. Any invocation of, e.g., `self-insert-command' will be represented as a linear change, while anything that doesn't fit the linear mold is represented as keystrokes.

Well, but must "linear changes" are very easily covered by the keystrokes version (e.g., insertion of "abc" is just "abc"). Others are problematic (note that vim-mode's repeat system is keystroke based and linear changes have never been problems).

I am replying to both of your last two posts. My point is that auto-completion /is/ a linear change. In fact, disregard the word "linear" (probably the most abused word in the mathematical dictionary), and try to think of a series of completion commands as a kind of "bumbling typist": "Now, what is that word I want to type, 'abacus'? So, a-b-a-c-u-s [6 letters]. No, that's not right, although the word does start with 'ab': Backspace, Backspace, Backspace, Backspace [4 times]. How about, hm, 'abbreviation'? b-r-e-v-i-a-t-i-o-n. No, that's not exactly right either ..." It's not as intimidating now, right? Since we only look at the actual /changes/, there are no nasty time delays or special keys to reckon with. It's just typing. This "bumbling typist" only presses letter keys and Backspace; he can either insert text from the left to the right, or delete text from the right to the left. Thus, he doesn't know about arrow keys (he cannot move the cursor directly). Nor does he press Delete (he can only delete text in one direction). If all vi users were "bumbling typists", we wouldn't need a keystrokes-based system; recording the inserted text would be sufficient. To compare the two approaches, Vim doesn't repeat paired delimiters ("(" and ")" around point), but it handles completion well. The keystrokes-based approach does repeat paired delimiters, but it goes "too far" with completion. So, could we combine their strengths and make the system fall back on old-fashioned insertion when the user calls a completion command? It seems we can, by monitoring changes from `after-change-functions'. If the current buffer changes /could/ have been performed by a "bumbling typist", they are recorded as insertion. If not, the keystrokes are recorded instead. Things the "bumbling typist" can do: * Insert text directly before point * Delete text directly before point * Insert text, delete some of it, and insert more (completion) Things the "bumbling typist" /cannot/ do: * Move point * Change the buffer * Save the buffer * Delete text after point * Insert text after point * Delete text not directly before point * Insert text not directly before point If the command does something from the first list and /not/ anything from the second list, then (and only then) it's recorded as insertion. Otherwise, it is recorded as keystrokes. So, we should automatically get completion in the first category, and paired delimiters in the second.

...

...

the first three arguments should be (BEG END &optional TYPE), and maybe a fourth one for the register. The define macros should let one specify further arguments, though.

Perhaps the order of arguments depends on the order of the interactive specification.

Yeah, and the `define-operator' macro should let one add code to that specification (not replace it, just augment it). Actually, `vimpulse-define-operator' allowed for this, just not in a very obvious way. If the body contained an `interactive' form, the return value of that form (which must be a list) would be concatenated with the calculated values for BEG and END. E.g., (vimpulse-define-operator vimpulse-insert (beg end &optional arg) "Insert before point." (interactive (list current-prefix-arg)) ...) This would roughly expand to the following function definition: (defun vimpulse-insert (beg end &optional arg) "Insert before point." (interactive (append (vimpulse-calculate-beg-and-end) (list current-prefix-arg))) ...) There might be a better syntax for this, but the functionality is there.

...

Btw, what about commands like , and ;. The do not have a specific type. Their type may be either explicit or implicit depending on the previous search command.

Good point. "}" is another motion which, depending on the circumstances, may use the `line' type instead of `exclusive'. I ended up with a global variable for the current type, which is initially set when the motion is executed, but may also be changed by the motion itself. The final value is used for the type when the buffer positions are normalized.

...

...

There is one case where I cannot see any other solution than a blacklist, though: normalizing Emacs' region to the Visual selection. We have to do this /before/ the next command is called (so we can't inspect its behavior), and we don't want to do it if the next command is a motion.

I do not understand. You can inspect the command in a pre-command hook and normalize the region accordingly or what do you mean? Of course, this only works if the commands use the interactive statement nicely.

You can inspect the command, but you can't really know what it is going to do. You could check out `interactive-form' to get a clue (which I tried), but the problem is that the form turns into mush if the function is compiled. Also, the command's behavior may be contingent on the activation of the region (e.g., skeletons). Consequently, there is no reliable way to identify region commands, at least not before the command is executed. It /is/ possible to identify all motions defined with the `define-motion' macro, though. :)

...

Perhaps we should just give define-motion not only the ability to define new motions but also the extend old motions with the necessary meta-data?

Yes. Very much yes. One should, for example, be able to specify that Emacs' own `next-line' and `previous-line' commands have a type of `line' (which I have, since I use them for "j" and "k" in my .emacs).

...

Well, I never cared about cl dependencies, I often used macros like when, unless, lexical-let and defun*. Imo cl is not really a bad dependency at least at compile time.

`when' and `unless' are provided by vanilla Elisp (in subr.el), cl.el just redefines them. `defun*' is for keyword arguments, which we'll probably not need. :) `lexical-let' provides lexical scope, which I've never used in Vimpulse. (I've only had one use for it: in my test framework, I needed to evaluate forms defined outside the evaluating function; regular local bindings could overwrite the form. What do you use `lexical-let' for? Closures?)

...

When you press "dw", "d" calls `delete', `delete' reads "w", "w" is bound to `forward-word', `forward-word' is executed to get buffer positions, and the positions are passed to the operator's arguments. Then the operator's body is executed.

It's just the difference to how vim-mode combines motions and operators. In vim-mode operator-pending is just a state as any other state, insert, normal or visual. When an operator is executed, the command's body is not executed immediately. Instead it is remembered for later use, then vim-mode switches to operator-pending mode and gives the control completely back to Emacs. When a motion is executed in operator-pending mode the stored operator is finally executed along with the motion. Similar things happen when a register is selected. In this case the register is stored and then vim-mode returns to normal mode and proceeds as usual.

If I understand vim-mode correctly, the motion is read and executed in another iteration of the command loop. In Vimpulse, everything happens immediately: "w" is read from the keyboard, the corresponding command is found with (key-binding "w"), and then it is executed. I ended up building a little "keypress parser" for handling counts as well. This is able to handle things that Emacs' command loop actually can't, such as "da2w". See below (search for `vimpulse-keypress-parser'): http://gitorious.org/vimpulse/vimpulse/blobs/master/vimpulse-operator.el Vimpulse does have a brief Operator-Pending state, which is entered before `key-binding' is called. When the motion is finished executing, the state promptly switches back to Normal. Thus, everything is done in a single iteration of the command loop.

Little question: do you really use Emacs's default forward-word? I ask because it behaves differently than Vim's.

One thing that annoyed me with Emacs' default command set is that there is no command for moving to the /beginning/ of the next word. With the exception of line movement, I don't use Emacs' commands. However, we may reuse Emacs' /concept/ of sentences and words. The `sentence-end-double-space' variable, for example, influences how sentences are understood. Should we honor such things, or should we invent our own system from scratch? -- Vegard

On 2011-03-01 16:57 +0100, Frank Fischer wrote:

Am Dienstag, 1. März 2011 schrieb Vegard Øye:

When some of those completion-dialogs pops up, the user can keep on typing or use the cursor-keys or whatever to select possible completions. How do you realize that these commands should *not* be recorded although they may look like, e.g., a motion (I just don't know how these packages really work). If this packages inserts text behind point it will drop you in the second case, but then the sequence of keys just does not represent what has been inserted (the sequence may be something like [a b b <down> <down> <up> c d <enter>].

What kind of command it is can only be determined in `post-command-hook', after the command has done all its stuff -- completion or otherwise. `after-change-functions' collect info while the command is executing, and then the hook uses that info to decide.

Another point why I would prefer key-sequence as default in all cases is that sometimes insertion triggers other changes, too, like indentation. I this case repeation of key-sequences does what you want (indent according to where the repeation took place) while direct insertion of text may fail. This may depend, in this example, on whether the additional spaces are inserted just before point.

True. It's just a heuristic, and it probably will have to be supplemented with some hard-coding ("this is an indentation command") as more complex use cases emerge. I don't see a way around the dreaded blacklist -- just ways to (hopefully) make the blacklist shorter.

In fact, is there a way to detect motion of point? Again another pre-/post-command-hook hack or something better? I ask because in some situations a certain motion or sequence of motions will actually change point while in other situations point will left unchanged (I think of motions like "tx" which does not move point if we are right before an x - okay we are in insert mode but again such situations may happen, e.g., end-of-line by leave point unchanged in one situation but when we repeat the keq-sequence elsewhere it would not).

Well, the rule does say that keystrokes are the default representation. "tx" will be recorded as keystrokes because it doesn't trigger an insertion. A command will only be recorded as insertion if it inserts or deletes text /and/ it doesn't do something which cannot be represented as an insertion. Otherwise, it's all keystrokes.

It seems to me that the key-sequence approach is almost always the right way. If there weren't those nasty completion things ... ;)

I agree. We might be overanalyzing this problem. Maybe the blacklist is a tolerable solution after all, because the odd completion command is all that needs to be listed.

...

(vimpulse-define-operator vimpulse-insert (beg end &optional arg) "Insert before point." (interactive (list current-prefix-arg)) ...)

Perhaps we could extend the string-variant by further chars understood by define-operator? Would make the stuff more readable.

Seems like a bit of a job, judging by the length of `interactive's docstring. I'm not sure it would be worth the effort.

But I wonder if it is possible to switch to operator-pending, then call read-key-sequence (or something like this), then switch back and continue as before. Of course, before switching one has to read the numeric prefix but this should be possible, I think.

The problem is handling prefix arguments. If you have two counts, like in "2d2w", the common heuristic is to multiply them together, effectively giving "d4w". But with Emacs' `digit-argument', it is difficult to get "in between" the motion count and the motion and say, "But multiply by 2 first!" Hence, `vimpulse-keypress-parser' reads a motion and its count without executing them, then the motion count is multiplied with the operator count (if any), and finally the motion is executed. Each step is disentangled from the others.

You use :contract and :expand ... I wonder something like the following would better reflect what we need:

evil:region "returns BEG and END of the real region covered by motion"

evil:line-region "returns BEG end END of the lines covered"

But I think you have a special use case where the suggestion above does not fit, right? So what do you expect from :expand and :contract, especially, why do you need the inverse transformation? It should be equally easy to store the original pair of BEG and END so the inversion should not be necessary.

Good question. On the face of it, there doesn't seem to be much use for :contract, since you can just remember the previous positions. (Indeed, the `line' transformation is irreversible, so in its case you /must/ remember them.) The reasons I want contraction are more subtle and based on experiences with "gv". How do you save a Visual selection for later use? Well, you can use markers to store the boundaries. But what boundaries do you store -- the unexpanded ones, or the expanded? Now we have two approaches: * Remember the unexpanded buffer positions of point and mark, along with the Visual submode; these three values define the selection. In other words, just put everything exactly where it was the last time. * Remember the /expanded/ positions of point and mark, plus the submode. Contract the positions as you re-enable the selection. I call these the "intuitive approach" and the "other approach", respectively. As you've guessed by now, the intuitive approach has some weaknesses. Since point and mark "expand" to an inclusive selection, the largest position is one less than the end of the selection. In other words, one of the markers is actually /inside/ the selection. What would happen time and again when I applied something slightly destructive on the selection was that the marker would get pushed out of it. Then, when I did "gv", the selection would be one character larger than before. ("d" is /too/ destructive to demonstrate this, while "u"/"U" is at the opposite end of the spectrum since they do everything character by character. You need something "in between", preferably a command which inserts and deletes here and there -- like `comment-region'. It's a subtle bug.) So what the "other approach" does is to get the markers out of the way by expanding them first. That way, a destructive command won't affect the markers, since they coincide exactly with the boundaries of the text the command is working on. Of course, this approach does require the inverse of expansion -- contraction -- afterwards. And in the case of `line' you actually cannot contract, so you'll have to fall back on remembering the original positions. Still, this solved the problem. Another thing: is using ":" as a prefix separator good for anything in Emacs Lisp? I haven't seen this practice elsewhere; all other Emacs packages separate their prefix with a hyphen (as suggested in the Emacs Lisp Reference Manual). -- Vegard

On 2011-02-28 10:02 +0100, Frank Fischer wrote:

Am Samstag, 26. Februar 2011 schrieb Vegard Øye:

...

One solution may be to define the repeat history as a collection of keystrokes and buffer insertions, e.g.,

([a b c] "inserted text" [d e])

Most commands are remembered as keystrokes (vectors), but certain ones, like `dabbrev-expand', are remembered as insertions (strings) instead.

I wonder if it is that easy. What kind of repeation is intended? Sometimes you want to repeat the inserted text, but another person wants to repeat the expansion context sensitive.

No, it is definitely not easy. :) But it might help to extend the "inserted text"/"actual keystrokes" split to the changes themselves. On one hand, there are "linear changes", which are changes for which one doesn't need a keystrokes-based system at all (the insertion-based approach of Vim would work equally well). Examples: * Insert two letters at (i.e., before) point * Insert three letters and delete the last two * Insert four letters, delete the last three, and insert five * Delete the last two letters Since we may delete more than we insert, a mere string won't capture the concept entirely; we'll need an offset as well. Thus, the changes above may be represented as: * ("aa" . 0) * ("aaa" . 2), or just ("a" . 0) * ("aeeeee" . 0), which may be seen as the "sum" of ("aaaa" . 0), ("" . 3), and ("eeeee" . 0) * ("" . 2) Now for some non-linear changes: * Insert "(" at point and ")" after point * Delete "(" before point and ")" after point These cannot be represented like above (and, indeed, Vim's repeat fails miserably when using the extension AutoClose.vim). Hence, for such changes we must default to keystrokes. (Vim's repeat actually fails even in trivial cases: try to repeat "i<Del><Del><Del>". It is not pretty.)

First using markers does not work in all cases, only if point is placed right after the inserted text. Other insertion packages may place point, e.g., somewhere in middle of the inserted text. Propably it's better to use after-change-functions.

If it is possible to reliably distinguish linear changes from non-linear ones in `after-change-functions', we can choose the appropriate representation accordingly. Any invocation of, e.g., `self-insert-command' will be represented as a linear change, while anything that doesn't fit the linear mold is represented as keystrokes.

And sometimes it may be quite difficult to find out which key-sequence to drop - I thing of auto-completion packages where during completion the user executes several commands and only few of them (the final RET?) really inserts some text, and this completions are often triggered by a timer, not by an explicit command.

Provided the completion does work linearly -- i.e., it only changes the text immediately before point -- the keystrokes shouldn't matter.

I would prefer some good default rule which works in most cases (blacklists are okay for rare cases).

I agree; blacklists should be avoided if possible. For the rare, intractable cases, though, we could add some variables for overriding the default choice. These could then be adjusted in a `defadvice' of the troublesome command. (We don't actually have to use advice, of course; an internal data structure would work just as well.)

Motions in vim-mode are represented by the vim:motion struct. This struct contains the two buffer positions along with the intended motion type (inclusive, exclusive, linewise, block). It is this data-structure that is passed to operators. So the single motion argument for operators in vim-mode is rather equivalent to the two parameters BEG END in vimpulse plus the type of the motion.

I see. We might have use of such a structure when passing values around internally, but I think the commands themselves should do things "out in the open" -- i.e., the first three arguments should be (BEG END &optional TYPE), and maybe a fourth one for the register. The define macros should let one specify further arguments, though.

This can be done either implicit or explicit. Explicit means that the function body really creates the structure and returns it. This is usually the case for text-objects (they are just usual motions in vim-mode) or some special commands like , or ; which return motions of different types depending on the previous f F t T command. Implicit means that the function does not return an explicit vim:motion structure. In this case it is enough to move point.

Yeah, all of Vimpulse's motions and text objects are implicit. Text objects are implemented as selection commands (since in Visual state, they /are/ selection commands). I experimented with return values at one point, but didn't find much use for it.

I can think of defining operators with arguments (BEG END &optional (type 'char) register) and an apropriate interactive specification. This would enable them to behave like normal region-operations but gives the additional possibilities to pass further important information. But then there is the problem how to pass the additional arguments interactively. But it sounds as if you have a good idea how to do this ;)

I can think of two approaches: 1. The `with-register' command (bound to ") reads a register from the keyboard, reads an operator, reads a motion (if appropriate), runs the motion to get buffer positions, and passes the positions and the register to the operator. 2. The current register is stored in a global variable which is reset in `post-command-hook'. The `with-register' command sets the variable, reads an operator from the keyboard, and executes the operator with `call-interactively'. The operator, as part of its interactive behavior, sets its REGISTER argument to the global variable. Neither is very difficult to implement: the code for "converting" motions to buffer positions can easily be reused in `with-register'.

...

...

- What to do with non-vim commands and motions?

Well, we will likely have one macro for defining operators, and another for defining motions. Both can be set up to index all defined operators/motions for later lookup, which will give us some information to go on.

As above, this would require a blacklist and have to keep the list up-to-date. And other packages may introduce new motions. So in all cases we need very good defaults so noone has to modify that list in almost all cases.

Yes. There is one case where I cannot see any other solution than a blacklist, though: normalizing Emacs' region to the Visual selection. We have to do this /before/ the next command is called (so we can't inspect its behavior), and we don't want to do it if the next command is a motion. So we can list either motions or non-motions, and normalize or don't normalize based on that. I went with listing motions because I figured the user is more likely to write a new region command than a new motion -- and Vimpulse does, after all, supply its own motions. If the documentation can convince the user to always use the `define-motion' macro for defining new motions, there shouldn't be much of a problem.

...

...

- How to switch states and how to represent states (usually as minor-modes, perhaps with some further meta-data like cursor, info-message, ...)

We'll have a macro for defining those too, of course. :)

Should we use standard minor-modes for this? I do this in vim-mode and it seems to be okay, but I wonder if there are any downsides.

I can't think of any. I think Viper solved this problem well enough.

But if I understand correctly you just want a function which changes point and mark and toggle transient-mark-mode to behave like a text-object ... or to enable visual-mode. Sounds reasonable to me although I'm always afraid of compatibility issue between Emacs regions and visual-regions.

Nah, this problem is pretty much solved, too. Vimpulse enables Transient Mark mode and activates the region when entering Visual state (and it tries to restore everything to their old values when exiting). This allows any region command to pick up on the selection, including commands whose behavior is conditional on the activation of the region, such as skeletons (defined with `define-skeleton').

...

I think the actual yank handlers should be specified by the type system, so that `line' has one handler, `block' has another, and so on. That way, we avoid hard-coding these associations into the "yank" command.

I prefer a type to be as simple as possible and to have as less additional functions and responsibilities as possible. The type should deal with its buffer positions. Not more. Anything else like handling of the visual-region or commands like yank/delete do not belong here.

Fair point. I agree that types should only specify a transformation and the reverse transformation if possible, plus perhaps a function for textual descriptions (for Viper-esque messages like "Deleted 2 lines"). A definition of the `inclusive' type could be: (define-type inclusive :expand (lambda (beg end) (list beg (min (1+ end) (point-max)))) :contract (lambda (beg end) (list beg (max beg (1- end)))) :describe (lambda (beg end) (let ((width (- end beg))) (format "%s characters" width)))) The words "expand" and "contract" refer to the fact that Vim's "inclusive" selections are larger than Emacs' "exclusive" ones. I'd prefer a more general pair of terms, though: maybe "transform"/"inverse-transform", or "transform"/"inverse", or "normalize"/"reverse", or something. Suggestions? Oh, and you /can/ step through this thing with Edebug, which is actually quite good at handling macros provided one tells it how (with `declare'). But then there's cl.el. Vimpulse used cl.el when I started out, but after various issues with `destructuring-bind' and Edebug (among other considerations), I decided to revert to "vanilla" Emacs Lisp. Would this be inconvenient to you? Are there some cl.el macros which would be valuable when solving the kinds of problems we're discussing here, or can we avoid this dependency?

Other commands require additional character arguments like "r" or motions like "f". Of course those commands can read that character themselves. But I prefer passing the character as an argument because then this command can also be used non-interactively (useful, e.g., for commands like , and ;).

I agree. "f" should call `read-char' in its `interactive' specification, while the body of the function should just concern itself with finding CHAR in the buffer. This is very readable.

...

...

The main problem is to cover all those funny special cases that arise (what happens at the end of the buffer, on empty lines, ...) - most bugs found in vim-mode are of this kind.

This can be tested by executing a keyboard macro in a temporary buffer.

Note that executing a keyboard macro makes testing certain aspects very difficult, namely those that behave differently when executed in a keyboard-macro or not. I think of the repeating system which must take care of keyboard-macros.

Yeah ... hm. Maybe we could invent our own variant of `execute-kbd-macro' for testing purposes.

I think errors should never be suppressed. The operator should just be canceled but the error should be shown. It is always dangerous to suppress errors (why does this not work? There is no error, right?)

Note that there are other error-symbols in use like end-of-buffer and the like. I think we should use them whenever possible. But should we use special error-symbols for errors that occur in our code?

We could always create our own function for error reporting. For example, less serious errors could be logged in a hidden buffer (starting with a space), whence they're brought forth by calling a special command. This could prove useful when helping users: "Please do `M-x show-error-log' and post the output". The logging level is the same, but the "noise level" is adjustable. I don't have strong feelings about this. As long as I don't get thrown into the debugger when I try to move past the end of the line, I'm okay. :)

Yes it is definitely a difference for commands to work on screen-lines or visual-lines. Note that deleting a visual line may have funny results because the other lines may change a little bit, too (long words deleted which caused line-breaks).

Yeah, I experimented a little bit with selecting screen lines in Visual state, but called it off because of pasting issues. It might work if one had a special yank-handler for such lines.

Very good, texinfo allows for nice inline documentation via :help ;)

Good point. Perhaps :help could open our documentation, while :info would go to the main Info index? -- Vegard

On 2011-02-28 10:02 +0100, Frank Fischer wrote:

"evil" really gave me a smile ;) I thought of something like "VeM" but have no good idea for its interpretation. "vimu" sounds good, too.

So it's "evil" versus "vimu", I guess. I don't have a clear favorite, so I'll try to list some pluses and minuses with both ... evil: + clever ("extensible vi layer") + fits in with the "vi vi vi is the devil's editor" mythology - may be difficult to search for - "X is evil" often means "X is broken" in the software world vimu: + searchable + multiple interpretations ("vi/Vim emulation") - maybe too Vim-like to suggest originality -- Vegard

On 2011-03-01 16:57 +0100, Frank Fischer wrote:

Well, whatever we decide for all those points above, I think it's time to start (if we want, I do ;)).

To summarize, I think the follow things have to be done first:

- create gitorious project

We'll need a name, then. ;) I vote for vimu. Tim Harper votes for evil. Christoph Lange votes for evil. That means the name will be decided by your vote. I don't feel strongly either way. Choose what /you/ think is best. :) -- Vegard

On 2011-03-01 21:03 +0100, Frank Fischer wrote:

Then it's evil ;)

And so it is: http://gitorious.org/evil :) As a starting point, I have uploaded parts of the development code I showed you earlier. Please verify that you have commit access by following these steps: * Go to Dashboard -> Manage SSH keys > Add SSH keys and upload the right file from ~/.ssh/. (If you don't have a SSH key, you can generate one with ssh-keygen -C "[email protected]".) * Set up your Git name and e-mail: git config --global user.name "Your Name" git config --global user.email "[email protected]" * Download the repository: git clone [email protected]:evil/evil.git You'll need the password for your SSH key, of course. (Non-committers can do git clone git://gitorious.org/evil/evil.git.) * Make a change. You can commit it with the command: git commit -a That will bring up an editor for the commit message. It should consist of a brief summary in the imperative (no period), followed by a blank line, followed by one or more additional sentences as necessary. For example: Add license The license is GNU GPLv2. Or anything. * Push your changes with: git push If I have pushed in the meantime, you will get an error, and must weave those changes into your own. You can do: git pull This will download my changes and merge them into yours. Alternatively: git pull --rebase This will just put your changes on top of mine. The latter is preferred if the changes are trivial and/or there is no overlap, since it retains a linear history. If there is overlap, you must merge and sort out any merge conflicts. See http://progit.org/book/ch3-2.html#basic_merge_conflicts for details. The first three chapters of that book should sort out the main Git/Mercurial differences. I hope the initial hurdles are surmountable. Let's get started! -- Vegard

On 2011-03-02 14:11 +0100, Frank Fischer wrote:

I wonder if gitorious has something like an issue tracker, which is usually quite useful?

Unfortunately not (yet). There is a field for linking to an external tracker, but no house implementation. If you just want a place to keep track of possible future improvements, I suggest using the wiki: http://gitorious.org/evil/pages/Home Where you'll also find an attempted logo for Evil. :) -- Vegard

On 2011-03-02 18:26 +0100, Tim Harper wrote:

This one makes 'vi' stand out more, I think

Good point, but I'd like the inverted letter to symbolize the cursor. Another way of making "vi" stand out more might be to move the devil's tail to the "i" letter, so that it becomes the "dot". That would be reminiscent of the fact that Vim is occasionally spelled "ViM": http://gitorious.org/evil/pages/LogoIdeas I've put up all three versions for comparison. Voice your opinions. -- Vegard

On Mon, Feb 28, 2011 at 2:02 AM, Frank Fischer <[email protected]> wrote:

I really like the idea of making operators as close to usual emacs commands as possible. This is certainly possible for character motions, but how do you tell which kind of motion the region belongs to? Call different commands depending on the type? Modify point and mark? The latter is possible for character and linewise motions, but what about block? Furthermore modifying point and mark may be problematic. Several commands have rules where to place point after execution depending on the original position of point (before modification to respect the visual region).

I really like this as well. With the original vimpulse (before Vegard got involved? My first experience with vimpulse was from a version posted to the Emacs wi) you had two separate regions, one for vimpulse visual mode and one for emacs.  This meant that emacs commands couldn't cooperate with vimpulse commands, and I personally found that totally repulsive.  I really like how vimpulse's visual mode operates now, it's wonderfully implemented, despite a few minor shortcomings. Perhaps it's just me, but if it's a choice between interoperability with Emacs at the cost of 97% compatibility with vim, and 100% compatibility with them at the cost of reduced interoperability with Emacs commands, I would prefer sacrificing the vim compatibility. If 100% vim compatibility were important to me, I would use vim.  I don't use vim because I enjoy what emacs provides, and prefer to use Emacs commands and vim commands interchangeably.  I recognize that many people consider me a freak for doing so, but I just use whatever makes sense to me at the time.

Am Dienstag, 1. März 2011 schrieb Vegard Øye:

On 2011-03-01 18:41 +0100, Tim Harper wrote:

...

Perhaps it's just me, but if it's a choice between interoperability with Emacs at the cost of 97% compatibility with vim, and 100% compatibility with them at the cost of reduced interoperability with Emacs commands, I would prefer sacrificing the vim compatibility. If 100% vim compatibility were important to me, I would use vim.

Well put. I totally agree.

I also agree. But I AM one of those who use both, vim and emacs, vim for quick changes and emacs for the real stuff, and sometimes even trivial differences become, well, not problems but nconveniences, if your fingers decide on their own which keys to press. For example, I once got a bug-report, because vim-mode placed cursor after a search-word command * after the found occurence (inherited from isearch that time) while vim puts the cursor at the beginning of that word. The reporter was used to type * again to repeat the search which did not work in vim-mode because the cursor was not ON the word. I never realized that this was problem because I'm used to type n to repeat the search - but everyone has its own workflow and so this was a valid bug-report. I completely agree that we should not focus on 100% vim compatibility (even sometimes Vim does things wrong IMO). We should just be prepared that sometimes incompatibilities or inconveniences may arise that are no problem for one user but are important for the other (and this with a good reason as the example above illustrates). So if the effort is not too big we should try to be as close as possible to vim without overemphasizing this goal. In any case, user feedback is important to receive information where those unexpected differences are. Bye, Frank

On 2011-02-26 11:38 +0100, Christoph LANGE wrote:

IIRC C-f in Vimpulse does "page down" by default, whereas in vim-mode it does something else. C-e in Vimpulse doesn't map to any vim-like feature, but to end-of-visual-line, whereas in vim-mode it does something else (don't remember, maybe copying from above/below in insert mode?).

This is mainly a question of whether one wants Insert state to be Emacs. Viper has an option: `viper-want-emacs-keys-in-insert'. When non-nil, most Insert state bindings are disabled, so that Emacs' bindings become available. It's a useful option to have, but since Vim is the emulation target, I think it should be disabled by default. -- Vegard