Vimpulse and vim-mode

[Frank Fischer]

Am Samstag, 26. Februar 2011 schrieb Vegard Øye:
> On 2011-02-25 22:20 +0100, Frank Fischer wrote:
> > Gitorious should be okay.
>
> Then Gitorious it is! :)

yuk ;)

> A big benefit of recording the keys is that it removes the need for
> repeat code in the commands themselves. For example, I have written
> some custom insertion commands for dealing with Lisp code (e.g.,
> "Insert after S-exp"); if I want them to repeat correctly under
> Viper, I have to edit the repeat history from within the commands,
> which is /extremely/ cumbersome. By contrast, a repeat system which
> monitors keystrokes and the current state would simply take note that
> I have entered Insert state, and update the repeat history
> accordingly.
>
> As you point out, some keystrokes are troublesome: "M-/", which calls
> `dabbrev-expand', may expand to different words in different places
> in the buffer. One solution may be to define the repeat history as a
> collection of keystrokes and buffer insertions, e.g.,
>
>     ([a b c] "inserted text" [d e])
>
> Most commands are remembered as keystrokes (vectors), but certain
> ones, like `dabbrev-expand', are remembered as insertions (strings)
> instead. The repeat system would use a "blacklist" of unsafe commands
> like `dabbrev-expand'; if the current command is listed, the system
> creates a marker at point (in `pre-command-hook'), lets the command
> execute, and then (in `post-command-hook') extracts the text between
> point and the marker, which goes into the repeat history like above.

I wonder if it is that easy. What kind of repeation is intended? 
Sometimes you want to repeat the inserted text, but another person 
wants to repeat the expansion context sensitive. I really have to think 
about a good compromise (of course dabbrev-expand may yield different 
results - but are they intendend? does this happend regularly or is it 
just a rare case?).

Some further comments. First using markers does not work in all cases, 
only if point is placed right after the inserted text. Other insertion 
packages may place point, e.g., somewhere in middle of the inserted 
text. Propably it's better to use after-change-functions. Second in 
general I do not like blacklists. The reason is that there are many 
Emacs packages out there each may define its own commands. Furthermore 
each new Emacs version could introduce new commands and we have to keep 
the blacklist up-to-date. Of course, each user can modify this list and 
adjust it to the packages she uses. But I think this may be cumbersome 
for the average user. And sometimes it may be quite difficult to find 
out which key-sequence to drop - I thing of auto-completion packages 
where during completion the user executes several commands and only few 
of them (the final RET?) really inserts some text, and this completions 
are often triggered by a timer, not by an explicit command.

I would prefer some good default rule which works in most cases 
(blacklists are okay for rare cases). This does not mean I have a good 
solution for these problems just a few things to consider (and I often 
run into the auto-completion problem in programming, so this is not a 
rare case).


> > - undo, vim usually undos everything done in insert-mode at once
> > while emacs undos insertion in steps.
>
> This is not that difficult in itself, but becomes more complex when
> loading a custom undo package like undo-tree.el. (I would really like
> to use undo-tree.el in place of redo.el; it features a graphical
> interface which totally outclasses Vim's :undolist.)

I never used :undolist so I can't say anything useful to it. For me redo 
has always been okay, so this is up to you ;)

>
> There is a case for making Vim-like "bulk undoing" toggleable:
> Emacs' fine-grained undoing can be valuable in some circumstances.
> The default should be Vim-like behavior, of course.
>
> > - catching of commands and motions. In each state the execution of
> >   commands and motions has different effects and therefore some
> >   special function has to be executed in each mode to deal with
> >   commands and motions. In general I see two possibilities how this
> >   special function can be executed. Either one hooks into pre-
> > and/or post-command hooks and checks whether the command/motion
> > about to be executed is a Vim operator and in this case do whatever
> > is required according to the current state. Or each command and
> > motion is defined in a special way in order to call the state
> > specific handler when it is executed. vim-mode uses the latter one
> > because I try to avoid pre- and post-command hooks if possible.
>
> Ah, the relationship between operators and motions. This is an area
> where Vimpulse and vim-mode differ somewhat. In vim-mode, as I
> understand it, motions are passed to operators by parameter (or the
> operator reads a motion from the keyboard when called interactively).
> The operator executes the received motion to determine what text to
> act on. The resulting pair of buffer positions is used internally.

That's wrong. Vim-mode almost behaves like what described below. Perhaps 
it's time to tell you how the things work in vim-mode (and I think this 
is not too bad, but certainly can be improved).

Motions in vim-mode are represented by the vim:motion struct. This 
struct contains the two buffer positions along with the intended motion 
type (inclusive, exclusive, linewise, block). It is this data-structure 
that is passed to operators. So the single motion argument for 
operators in vim-mode is rather equivalent to the two parameters BEG 
END in vimpulse plus the type of the motion. Usually operators use the 
motion type to behave differently (but this is not required). If we had 
CLOS-like methods one would propably dispatch on the motion type.

A motion in vim-mode is function that returns a vim:motion structure. 
This can be done either implicit or explicit. Explicit means that the 
function body really creates the structure and returns it. This is 
usually the case for text-objects (they are just usual motions in 
vim-mode) or some special commands like , or ; which return motions of 
different types depending on the previous f F t T command. Implicit 
means that the function does not return an explicit vim:motion 
structure. In this case it is enough to move point. Then the magic 
within vim:defmotion ensures that a vim:motion structure is 
automatically created depending on the default type of the motion and 
the movement of point. Implicit is therefore enough for almost all 
simple motions what makes them one-liners. Note that from the caller's 
point of view it makes no different to call an implicit or explicit 
returning motion - the caller always receives the vim:motion structure.

When executing an operator the operator is not executed immediately. 
Instead the execution of its body is postponed until a motion has been 
selected. When the motion has been selected, the handler of 
operator-pending-mode first executes the motion to create the 
vim:motion structure and then passes this structure to the operator 
which is executed afterwards. In visual-mode the vim:motion structure 
is not created by a motion but w.r.t. to the current visual selection.

All in all, the representation of a motion is very simple, just three 
values, BEG END and TYPE.

I decided to use keyword arguments because in vim-mode commands and 
motions may take a different set of arguments. E.g., a command may take 
 - count
 - motion
 - register
 - additional argument (char, text, file, buffer, ...) whereas the 
latter are only usefull for ex-commands
 - force (only for ex)
Also note that in vim-mode each command can be bound in ex-mode AND 
normal-mode/visual-mode. In my philosophy ex-mode is just another mode 
with own keybindings, although they are read in a different way.

> In Vimpulse, the buffer positions /are/ the parameters (BEG and END).
> Operators work like region commands (i.e., (interactive "r")) in that
> they either receive the positions from the caller, or figure them out
> for themselves when called interactively. (Vimpulse also possesses
> some facilities for "passing a motion to an operator", which executes
> the specified motion and feeds its range to the operator. This is
> only used by the repeat system, though, and it would be totally
> superfluous if a keystroke-based system was in place.)

I really like the idea of making operators as close to usual emacs 
commands as possible. This is certainly possible for character motions, 
but how do you tell which kind of motion the region belongs to? Call 
different commands depending on the type? Modify point and mark? The 
latter is possible for character and linewise motions, but what about 
block? Furthermore modifying point and mark may be problematic. Several 
commands have rules where to place point after execution depending on 
the original position of point (before modification to respect the 
visual region).

I can think of defining operators with arguments 
  (BEG END &optional (type 'char) register)
and an apropriate interactive specification. This would enable them
to behave like normal region-operations but gives the additional 
possibilities to pass further important information. But then there is 
the problem how to pass the additional arguments interactively. But it 
sounds as if you have a good idea how to do this ;)

> For the sake of reusability, I recommend defining operator commands
> in the same way as region commands: with two parameters, BEG and END,
> and an `interactive' specification for determining their values by
> reading a motion from the keyboard.
>
> > - What to do with non-vim commands and motions? There should be a
> >   simple rule so they work with vim in most cases as expected, but
> > if no meta-information is provided a non-vim command may always be
> > problematic (e.g., in operator-pending mode).
>
> Well, we will likely have one macro for defining operators, and
> another for defining motions. Both can be set up to index all defined
> operators/motions for later lookup, which will give us some
> information to go on. By the way, I also think that all of Emacs'
> standard movement commands should be regarded as motions.

As above, this would require a blacklist and have to keep the list 
up-to-date. And other packages may introduce new motions. So in all 
cases we need very good defaults so noone has to modify that list in 
almost all cases.


> We thus get a large list of commands which move the cursor, and can
> restrict Operator-Pending state to those commands. Alternatively, we
> can maintain a "blacklist" of Emacs commands which should /not/ be
> executed. In any case, I don't think it's overly problematic if a
> nonsensical command should get called; any damage will be undoable,
> after all. It's probably better to err on the side of allowing too
> many commands than too few.
>
> > - What to do with commands that change the current buffer? This is
> >   problematic if some buffer-local variables are involved.
>
> Right, which means that the repeat system cannot be buffer-local.
> I think it would just be confusing if it was, anyway.

Of course it is not, but there may be other buffer-local variables. In 
vim-mode a tackle this problem by saving the new-buffer when a command 
returns, switching back to the original buffer, do the necessary work 
and switch to new buffer at the end. This seems to work quite well but 
can certainly be improved.

> > - How to switch states and how to represent states (usually as
> >   minor-modes, perhaps with some further meta-data like cursor,
> >   info-message, ...)
>
> We'll have a macro for defining those too, of course. :)
>
> I suggest that each state's toggle function be a command; e.g., in
> Vimpulse's development code, vi state defines the command
> `vimpulse-vi-state' for entering that state, emacs state defines
> `vimpulse-emacs-state', and so on. The command also has an optional
> argument which, if negative, disables the state; this is used
> internally when switching from one state to another.

Should we use standard minor-modes for this? I do this in vim-mode and 
it seems to be okay, but I wonder if there are any downsides.

> > - What is a motion? There are a few fundamental differences between
> >   buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs
> >   knows just the offset. The choice how to represent coordinates
> > can be crucial. And of course there is that funny newline
> > character, which is usually invisible in Vim except for empty
> > lines. Because a motion is a important concept for many parts
> > including operators and visual-mode.
>
> I think all of Emacs' regular movement commands should be valid
> motions. Furthermore, all of Emacs' selection commands should be
> valid text objects. In Vimpulse, "y C-x h" and "C-x h y" are equally
> valid ways of copying the whole buffer, and "d M-e" works just as
> well for deleting a sentence as does "d)".

Hm I think d M-e is okay besides the fact that a sentence is slightly 
different in Emacs and Vim (I think), but never mind. The first example 
seems to be very special case. But if I understand correctly you just 
want a function which changes point and mark and toggle 
transient-mark-mode to behave like a text-object ... or to enable 
visual-mode. Sounds reasonable to me although I'm always afraid of 
compatibility issue between Emacs regions and visual-regions. Is it 
easy to detect that a certain command behaves like this or is it 
another blacklist variant? ;)

> Given the above, the only thing which separates Emacs' movement
> commands from the other motions is their lack of a /type/. For
> example, the motion "j" has a type of `line', which means that "dj"
> deletes two whole lines. What a type is, then, is a way to transform
> two buffer positions to two other positions -- in this case, to two
> whole lines.
>
> >   Probably a motion is represented by some abstract data
> >   type and this data-type is an important part of the interface a
> > user must understand in order to write new motions and commands.
> > Therefore it should be well-defined from the beginning because it
> > cannot be changed easily afterwards.
>
> I don't think it needs to be very abstract. I would rather try to
> keep things from getting too abstract, actually. One of the design
> rules I laid down in Vimpulse's new "TODO" list was that it should
> require as few packages as possible, and not invent new stuff unless
> it's absolutely necessary.

I think we both have the same feeling of it (see above). In fact, 
vim:motion in vim-mode comes with a couple of functions which do 
exactly this (returning appropriate buffer positions w.r.t. the motion 
type).

> The way I understand motions, they are just regular movement command
> plus a type. The type can be stored as a symbol property, e.g.,
> (put 'motion 'type 'line). The type itself, of course, must also be
> defined; for this, a minimal system may be warranted, since both
> Operator-Pending state and Visual state use types. I've pushed a
> currently unused "type system" to "development":
>
> http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-typ
>es.el
>
> Now, it's a bit more involved than it has to be; the routines for
> reselecting (e.g., two lines anywhere in the buffer) are unnecessary
> given a keystroke-based repeat system. The advantages of a systematic
> approach is that it can be reused by Operator-Pending state and
> Visual state, making the overall code simpler. The only heavy lifting
> Visual state ends up doing (once initialized) is highlight the
> different types, which is rather trivial (with the exception of
> `block' ...).
>
> > In fact, there are only two (or three) difficult commands: yank,
> > delete and paste, but I hope the implementation in vim-mode is
> > quite good know (it uses yank-handlers for all three kinds of
> > yanking and also provides "yank-pop" stuff).
>
> Yes, this is exactly the right way to do it. I think the actual yank
> handlers should be specified by the type system, so that `line' has
> one handler, `block' has another, and so on. That way, we avoid
> hard-coding these associations into the "yank" command.

At this point I completely disaggree. I don't like the idea of coding 
behaviours of yank/delete and so on into the type system for two 
reasons. First of all, yank/delete should not be different from any 
other command. A special handling of them in the type system seems to 
be confusing to me. The other reason is that there may be even more 
commands that have different behaviour depending on the motion type, 
not just yank/delete, even commands that we cannot think of because we 
want *extensibility*. 

I prefer a type to be as simple as possible and to have as less 
additional functions and responsibilities as possible. The type should 
deal with its buffer positions. Not more. Anything else like handling 
of the visual-region or commands like yank/delete do not belong here.

I think the main difference here is that you want to stick with the 
standard interface of Emacs for operators (only BEG END) while I want 
to provide additional information (especially motion type). So in my 
world it would be the responsibility of the command to behave 
differently on different motion types. Of course then your commands 
need to be sensitive to the motion types - but you have to deal with 
this sensitivity anyway, nothing prevents you from that. The question 
is where. 

Of course it would be nice if operators also behave like usual Emacs 
commands when used in such a context. This was the reason of my 
suggestion above (BEG END &optional type ...).

In fact, what I would need is a dispatch on the motion type, i.e., (yank 
beg end 'char) calls another function than (yank beg end 'line). I do 
not know of any such functionality in Emacs so propably it ends up with 
manual dispatch (case type (line (yank-line ...)))) and so on. I do not 
think this is too bad and certainly not more complicate than you 
suggestion. But I think it provides a better separation of code.

Note that it is okay for me to have certain functions with update visual 
region according to a motion type. But since visual-mode is the only 
place where this is required it should go there. Of course in a 
structured way.

> > What could be useful are frameworks or perhaps only some functions
> > which help defining more involved motions like text-objects. We
> > should provide helper-functions for typical text-objects like
> > parentheses, blocks, whatever. vim-mode (and vimpulse, too)
> > contains a few of those functions which are again independent.
>
> Agreed; we should have a macro for defining text objects, too.
>
> To recap, we now have the following define macros:
>
>     * `define-state'
>     * `define-type'
>     * `define-motion'
>     * `define-text-object'
>     * `define-operator'
>
> Do we need more than these?

Well there are also other commands that are not operators. E.g. paste 
requires at least two arguments, the count and the register. Other 
commands require additional character arguments like "r" or motions 
like "f". Of course those commands can read that character themselves. 
But I prefer passing the character as an argument because then this 
command can also be used non-interactively (useful, e.g., for commands 
like , and ;).

> >> http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-
> >>states.el
> >
> > I read your comments and they sound reasonable. Adding auxiliary
> > keymaps should not be that difficult. As I remember correctly,
> > local keymaps were the biggest problem in vim-mode, but major and
> > minor-mode specific should be relatively easy. Of course this is
> > very important and should be done as one of the first things.
>
> Actually, the current code already implements auxiliary keymaps,
> although I've not tested it thoroughly.
>
> > Note that vim-mode arranges keymaps in another hierarchy, too,
> > which allows to enable only few keybindings in certain major modes,
> > e.g., only movement and window-commands in info-mode (somehow like
> > viper-on-more-modes I think). The difference is that many
> > standard-keybindings are not available in this case, e.g.,
> > insert-mode.
>
> Yes, this is very sensible. Motions and text objects can be bound in
> their own keymap (or state), and that keymap is inherited by vi
> state, Visual state, and so on. Modes which do not work well in vi
> state, can come up in the motion state instead. Any shortcomings can
> be alleviated on a per-mode basis with mode-specific state bindings.
>
> >> Regarding test frameworks, I hear Christian Ohler's ert.el is now
> >> included with Emacs trunk, so we could go with that:
> >
> > I never used any of those libraries but we should try. The main
> > problem is to cover all those funny special cases that arise (what
> > happens at the end of the buffer, on empty lines, ...) - most bugs
> > found in vim-mode are of this kind.
>
> This can be tested by executing a keyboard macro in a temporary
> buffer. A lot of Vimpulse's early tests do this, e.g.,
>
>     (deftest test-change-undo
>       "Change a word and undo."
>       (execute-kbd-macro "wcwfoo^[u")
>       (assert-string=
>         (buffer-substring 1 51)
>         ";; This buffer is for notes you don't want to save"))

Note that executing a keyboard macro makes testing certain aspects very 
difficult, namely those that behave differently when executed in a 
keyboard-macro or not. I think of the repeating system which must take 
care of keyboard-macros.

> > This is another point which is currently very bad in vim-mode. How
> > to deal with errors, should there be a unique signalling scheme,
> > where to handle them, ...
>
> In "indirect" cases, like when executing a motion to get a text
> range, I think any errors should be suppressed. When executing the
> motions by themselves, however, we should signal any problems with
> `error'.

I think errors should never be suppressed. The operator should just be 
canceled but the error should be shown. It is always dangerous to 
suppress errors (why does this not work? There is no error, right?)

Note that there are other error-symbols in use like end-of-buffer and 
the like. I think we should use them whenever possible. But should we 
use special error-symbols for errors that occur in our code?

> > Btw, Emacs 23 contains visual-line-mode which is quite useful. But
> > most commands work on buffer lines not on screen lines. Should we
> > try to support visual-lines, too (optionally)? If yes this has to
> > be considered for motions and commands, especially for
> > operator-stuff, too, and we should think of it from the beginning,
> > because many commands have to be implemented differently for
> > visual-lines.
>
> Yes, we should definitely support screen lines; I have "j" and "k"
> bound to `next-line' and `previous-line' because they work better in
> visual-line-mode. In Vim, one uses "gj" and "gk" to move by screen
> lines. I think this is very inefficient and would prefer an option
> for the regular commands. It may of course be disabled by default.

Yes it is definitely a difference for commands to work on screen-lines 
or visual-lines. Note that deleting a visual line may have funny 
results because the other lines may change a little bit, too (long 
words deleted which caused line-breaks).

Furthermore block-motions do not really make sense for visual lines. I 
would definitely not use visual-line movements as defaults but document 
very well how it can be configured, because dealing with visual lines 
(as a user) may be very inconsistend.

> > Another thing somehow related to tests is documentation. We should
> > not only write developer-documentation but also user-documentation
> > right from the start. This should contain instructions for all
> > important concepts and also examples for new motions and commands
> > of all kind. Where should we put this documentation? EmacsWiki?
> > Hosting site? Perhaps as tex-info?
>
> Texinfo, definitely. I've had my share of arbitrary wiki syntax and
> never-ending comment blocks. A text file under version control is
> wiki-like enough for me.

Very good, texinfo allows for nice inline documentation via :help ;)


> > Hm, I think a new cool name would be good. I prefer short names or
> > at least names that allow short prefixes for function names ;)
>
> For the sake of brainstorming, I've grepped through /usr/share/dict
> for anything containing "vi". There isn't much, unfortunately, so
> I'll start with some acronyms comprising three letters:
>
>     * avi, evi (augmented/extensible vi)
>     * via, vie (vi augmented/extended)
>     * vil (vi layer)
>     * xvi (vi plus a cool letter)
>     * yvi (yet a(nother) vi)
>
> Things get slightly more descriptive when we allow four letters:
>
>     * evil (extensible vi layer; this almost works too well)
>     * nuvi (new (unlimited) vi)
>     * vibe (vi is beautiful?)
>     * vici (veni, vidi ...)
>     * vimu (vi(m) emulation; vim unlimited; vi made usable)
>     * vini (vi new and improved)
>     * viva (viva la vi!)
>     * yavi (yet another vi)
>

"evil" really gave me a smile ;) 
I thought of something like "VeM" but have no good idea for its 
interpretation. "vimu" sounds good, too.

Bye,
Frank