Vimpulse and vim-mode

[Vegard Øye]

On 2011-02-25 22:20 +0100, Frank Fischer wrote:

> Gitorious should be okay.

Then Gitorious it is! :)

> - repeating. Vim-mode uses keyboard-macro and those have to be
>   recorded. Recording them can be tricky if one keyboard macro calls
>   another one. Furthermore insert-mode is different because it
>   contains of several independent key-sequences. Using key-sequences
>   has some benefits but also some downsides, e.g., when lazy
>   completion packages are used

A big benefit of recording the keys is that it removes the need for
repeat code in the commands themselves. For example, I have written
some custom insertion commands for dealing with Lisp code (e.g.,
"Insert after S-exp"); if I want them to repeat correctly under Viper,
I have to edit the repeat history from within the commands, which is
/extremely/ cumbersome. By contrast, a repeat system which monitors
keystrokes and the current state would simply take note that I have
entered Insert state, and update the repeat history accordingly.

As you point out, some keystrokes are troublesome: "M-/", which calls
`dabbrev-expand', may expand to different words in different places in
the buffer. One solution may be to define the repeat history as a
collection of keystrokes and buffer insertions, e.g.,

    ([a b c] "inserted text" [d e])

Most commands are remembered as keystrokes (vectors), but certain
ones, like `dabbrev-expand', are remembered as insertions (strings)
instead. The repeat system would use a "blacklist" of unsafe commands
like `dabbrev-expand'; if the current command is listed, the system
creates a marker at point (in `pre-command-hook'), lets the command
execute, and then (in `post-command-hook') extracts the text between
point and the marker, which goes into the repeat history like above.

> - undo, vim usually undos everything done in insert-mode at once while
>   emacs undos insertion in steps.

This is not that difficult in itself, but becomes more complex when
loading a custom undo package like undo-tree.el. (I would really like
to use undo-tree.el in place of redo.el; it features a graphical
interface which totally outclasses Vim's :undolist.)

There is a case for making Vim-like "bulk undoing" toggleable:
Emacs' fine-grained undoing can be valuable in some circumstances.
The default should be Vim-like behavior, of course.

> - catching of commands and motions. In each state the execution of
>   commands and motions has different effects and therefore some
>   special function has to be executed in each mode to deal with
>   commands and motions. In general I see two possibilities how this
>   special function can be executed. Either one hooks into pre- and/or
>   post-command hooks and checks whether the command/motion about to be
>   executed is a Vim operator and in this case do whatever is required
>   according to the current state. Or each command and motion is
>   defined in a special way in order to call the state specific handler
>   when it is executed. vim-mode uses the latter one because I try to
>   avoid pre- and post-command hooks if possible.

Ah, the relationship between operators and motions. This is an area
where Vimpulse and vim-mode differ somewhat. In vim-mode, as I
understand it, motions are passed to operators by parameter (or the
operator reads a motion from the keyboard when called interactively).
The operator executes the received motion to determine what text to
act on. The resulting pair of buffer positions is used internally.

In Vimpulse, the buffer positions /are/ the parameters (BEG and END).
Operators work like region commands (i.e., (interactive "r")) in that
they either receive the positions from the caller, or figure them out
for themselves when called interactively. (Vimpulse also possesses
some facilities for "passing a motion to an operator", which executes
the specified motion and feeds its range to the operator. This is only
used by the repeat system, though, and it would be totally superfluous
if a keystroke-based system was in place.)

For the sake of reusability, I recommend defining operator commands in
the same way as region commands: with two parameters, BEG and END, and
an `interactive' specification for determining their values by reading
a motion from the keyboard.

> - What to do with non-vim commands and motions? There should be a
>   simple rule so they work with vim in most cases as expected, but if
>   no meta-information is provided a non-vim command may always be
>   problematic (e.g., in operator-pending mode).

Well, we will likely have one macro for defining operators, and
another for defining motions. Both can be set up to index all defined
operators/motions for later lookup, which will give us some
information to go on. By the way, I also think that all of Emacs'
standard movement commands should be regarded as motions.

We thus get a large list of commands which move the cursor, and can
restrict Operator-Pending state to those commands. Alternatively, we
can maintain a "blacklist" of Emacs commands which should /not/ be
executed. In any case, I don't think it's overly problematic if a
nonsensical command should get called; any damage will be undoable,
after all. It's probably better to err on the side of allowing too
many commands than too few.

> - What to do with commands that change the current buffer? This is
>   problematic if some buffer-local variables are involved.

Right, which means that the repeat system cannot be buffer-local.
I think it would just be confusing if it was, anyway.

> - How to switch states and how to represent states (usually as
>   minor-modes, perhaps with some further meta-data like cursor,
>   info-message, ...)

We'll have a macro for defining those too, of course. :)

I suggest that each state's toggle function be a command; e.g., in
Vimpulse's development code, vi state defines the command
`vimpulse-vi-state' for entering that state, emacs state defines
`vimpulse-emacs-state', and so on. The command also has an optional
argument which, if negative, disables the state; this is used
internally when switching from one state to another.

> - What is a motion? There are a few fundamental differences between
>   buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs
>   knows just the offset. The choice how to represent coordinates can
>   be crucial. And of course there is that funny newline character,
>   which is usually invisible in Vim except for empty lines. Because a
>   motion is a important concept for many parts including operators and
>   visual-mode.

I think all of Emacs' regular movement commands should be valid
motions. Furthermore, all of Emacs' selection commands should be valid
text objects. In Vimpulse, "y C-x h" and "C-x h y" are equally valid
ways of copying the whole buffer, and "d M-e" works just as well for
deleting a sentence as does "d)".

Given the above, the only thing which separates Emacs' movement
commands from the other motions is their lack of a /type/. For
example, the motion "j" has a type of `line', which means that "dj"
deletes two whole lines. What a type is, then, is a way to transform
two buffer positions to two other positions -- in this case, to two
whole lines.

>   Probably a motion is represented by some abstract data
>   type and this data-type is an important part of the interface a user
>   must understand in order to write new motions and commands.
>   Therefore it should be well-defined from the beginning because it
>   cannot be changed easily afterwards.

I don't think it needs to be very abstract. I would rather try to keep
things from getting too abstract, actually. One of the design rules I
laid down in Vimpulse's new "TODO" list was that it should require as
few packages as possible, and not invent new stuff unless it's
absolutely necessary.

The way I understand motions, they are just regular movement command
plus a type. The type can be stored as a symbol property, e.g.,
(put 'motion 'type 'line). The type itself, of course, must also be
defined; for this, a minimal system may be warranted, since both
Operator-Pending state and Visual state use types. I've pushed a
currently unused "type system" to "development":

http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-types.el

Now, it's a bit more involved than it has to be; the routines for
reselecting (e.g., two lines anywhere in the buffer) are unnecessary
given a keystroke-based repeat system. The advantages of a systematic
approach is that it can be reused by Operator-Pending state and Visual
state, making the overall code simpler. The only heavy lifting Visual
state ends up doing (once initialized) is highlight the different
types, which is rather trivial (with the exception of `block' ...).

> In fact, there are only two (or three) difficult commands: yank,
> delete and paste, but I hope the implementation in vim-mode is quite
> good know (it uses yank-handlers for all three kinds of yanking and
> also provides "yank-pop" stuff).

Yes, this is exactly the right way to do it. I think the actual yank
handlers should be specified by the type system, so that `line' has
one handler, `block' has another, and so on. That way, we avoid
hard-coding these associations into the "yank" command.

> What could be useful are frameworks or perhaps only some functions
> which help defining more involved motions like text-objects. We should
> provide helper-functions for typical text-objects like parentheses,
> blocks, whatever. vim-mode (and vimpulse, too) contains a few of those
> functions which are again independent.

Agreed; we should have a macro for defining text objects, too.

To recap, we now have the following define macros:

    * `define-state'
    * `define-type'
    * `define-motion'
    * `define-text-object'
    * `define-operator'

Do we need more than these?

>> http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-states.el
>
> I read your comments and they sound reasonable. Adding auxiliary
> keymaps should not be that difficult. As I remember correctly, local
> keymaps were the biggest problem in vim-mode, but major and minor-mode
> specific should be relatively easy. Of course this is very important
> and should be done as one of the first things.

Actually, the current code already implements auxiliary keymaps,
although I've not tested it thoroughly.

> Note that vim-mode arranges keymaps in another hierarchy, too, which
> allows to enable only few keybindings in certain major modes, e.g.,
> only movement and window-commands in info-mode (somehow like
> viper-on-more-modes I think). The difference is that many
> standard-keybindings are not available in this case, e.g.,
> insert-mode.

Yes, this is very sensible. Motions and text objects can be bound in
their own keymap (or state), and that keymap is inherited by vi state,
Visual state, and so on. Modes which do not work well in vi state, can
come up in the motion state instead. Any shortcomings can be
alleviated on a per-mode basis with mode-specific state bindings.

>> Regarding test frameworks, I hear Christian Ohler's ert.el is now
>> included with Emacs trunk, so we could go with that:
>
> I never used any of those libraries but we should try. The main
> problem is to cover all those funny special cases that arise (what
> happens at the end of the buffer, on empty lines, ...) - most bugs
> found in vim-mode are of this kind.

This can be tested by executing a keyboard macro in a temporary
buffer. A lot of Vimpulse's early tests do this, e.g.,

    (deftest test-change-undo
      "Change a word and undo."
      (execute-kbd-macro "wcwfoo^[u")
      (assert-string=
        (buffer-substring 1 51)
        ";; This buffer is for notes you don't want to save"))

> This is another point which is currently very bad in vim-mode. How to
> deal with errors, should there be a unique signalling scheme, where to
> handle them, ...

In "indirect" cases, like when executing a motion to get a text range,
I think any errors should be suppressed. When executing the motions by
themselves, however, we should signal any problems with `error'.

> Btw, Emacs 23 contains visual-line-mode which is quite useful. But
> most commands work on buffer lines not on screen lines. Should we try
> to support visual-lines, too (optionally)? If yes this has to be
> considered for motions and commands, especially for operator-stuff,
> too, and we should think of it from the beginning, because many
> commands have to be implemented differently for visual-lines.

Yes, we should definitely support screen lines; I have "j" and "k"
bound to `next-line' and `previous-line' because they work better in
visual-line-mode. In Vim, one uses "gj" and "gk" to move by screen
lines. I think this is very inefficient and would prefer an option for
the regular commands. It may of course be disabled by default.

> Another thing somehow related to tests is documentation. We should not
> only write developer-documentation but also user-documentation right
> from the start. This should contain instructions for all important
> concepts and also examples for new motions and commands of all kind.
> Where should we put this documentation? EmacsWiki? Hosting site?
> Perhaps as tex-info?

Texinfo, definitely. I've had my share of arbitrary wiki syntax and
never-ending comment blocks. A text file under version control is
wiki-like enough for me.

> Hm, I think a new cool name would be good. I prefer short names or at
> least names that allow short prefixes for function names ;)

For the sake of brainstorming, I've grepped through /usr/share/dict
for anything containing "vi". There isn't much, unfortunately, so I'll
start with some acronyms comprising three letters:

    * avi, evi (augmented/extensible vi)
    * via, vie (vi augmented/extended)
    * vil (vi layer)
    * xvi (vi plus a cool letter)
    * yvi (yet a(nother) vi)

Things get slightly more descriptive when we allow four letters:

    * evil (extensible vi layer; this almost works too well)
    * nuvi (new (unlimited) vi)
    * vibe (vi is beautiful?)
    * vici (veni, vidi ...)
    * vimu (vi(m) emulation; vim unlimited; vi made usable)
    * vini (vi new and improved)
    * viva (viva la vi!)
    * yavi (yet another vi)

With five letters or more, there are many proper names to choose from
(elvis is the name of a vi clone): alvin, levi, david, vidal, vince.
And we have:

    * anvil (a new vi layer; doesn't sound very light, though)
    * vibrant (the soothing, relaxing, vibrating editor)
    * vigil (vi greatly improved layer-something)
    * vital (vi is vital to, say, avoiding RSI ...)

More letters would be inconvenient; I want the name and code prefix to
be one and the same. (I use an abbreviated prefix in
viper-in-more-modes.el, and I hate the sight of it.)

Of the above, I rather like "vimu". It's short, it's sweet, and it can
be read as both "vi emulation" and "Vim emulation". Although I think
Vim sets the baseline for what a modal editor should support, it's not
the end of the road for me. A sufficiently extensible implementation
will enable the user to explore uncharted territory.

-- 
Vegard