Vimpulse and vim-mode

Fri Feb 25 22:20:18 CET 2011

On Fri, Feb 25, 2011 at 08:47:13PM +0100, Vegard Øye wrote:
> I would love to work on a common project. Getting all our users on a
> single "platform" will likely give us more user extensions, which is
> my main motivation for doing what I do -- the great thing about
> sharing code is that others share back, too. :)

agree

> How do you imagine we set up a common project? There is a couple of
> technical issues to sort out:
> 
> * Version control (Git vs. Mercurial)
> 
> The last time I checked (when I made the switch from SVN to Git),
> this list was rather Git-partial. (I actually argued for Mercurial
> then.) I have since grown to love how Git does a lot of things,
> like branching. Also, I have no experience with Mercurial.

I have no experience with git but I don't really care about the
concrete VCS. I use mercurial, well, because I've used it for several
years know and at that time git's documentation was poor. But IMO it
is just a tool to get a job done, both are almost equivalent in all
important aspects and both provide all the things we need. Most of the
time no one really needs the fancy stuff both provide but only
log/push/commit/merge and branch. And I can use git-reps with
mercurial anyway (and probably the other way around) ;)

> How about you?
> 
> * Hosting
> 
> If we go with Git, I suggest Gitorious as a hosting provider:
> it is project-oriented rather than person-oriented. I know
> of no project-oriented Mercurial providers.

Neither I do (perhaps sourceforge? There could be some
non-VCS-specific hosting providers). Gitorious should be okay.

> * Merging
> 
> Where do we start? What parts will benefit from a rewrite, and what
> parts can be included as-is? 

The most important part is obviously the core, which switches between
states and connects motions and commands. This core should be as small
as possible but flexible. The following things have to be considered
before we start because they have significant influence on the rest:

- repeating. Vim-mode uses keyboard-macro and those have to be
  recorded. Recording them can be tricky if one keyboard macro calls
  another one. Furthermore insert-mode is different because it
  contains of several independent key-sequences. Using key-sequences
  has some benefits but also some downsides, e.g., when lazy
  completion packages are used

- undo, vim usually undos everything done in insert-mode at once while
  emacs undos insertion in steps.

- catching of commands and motions. In each state the execution of
  commands and motions has different effects and therefore some
  special function has to be executed in each mode to deal with
  commands and motions. In general I see two possibilities how this
  special function can be executed. Either one hooks into pre- and/or
  post-command hooks and checks whether the command/motion about to be
  executed is a Vim operator and in this case do whatever is required
  according to the current state. Or each command and motion is
  defined in a special way in order to call the state specific handler
  when it is executed. vim-mode uses the latter one because I try to
  avoid pre- and post-command hooks if possible.

- Together with the point above the definition of commands and motions
  has to be clarified. This is the most important part because almost
  all user-extensions will be commands and motions and not special 
  modes or other funny things.

- What to do with non-vim commands and motions? There should be a
  simple rule so they work with vim in most cases as expected, but if
  no meta-information is provided a non-vim command may always be
  problematic (e.g., in operator-pending mode).

- What to do with commands that change the current buffer? This is
  problematic if some buffer-local variables are involved.

- How to switch states and how to represent states (usually as
  minor-modes, perhaps with some further meta-data like cursor,
  info-message, ...)

- What is a motion? There are a few fundamental differences between
  buffer-coordinates of Vim and Emacs. Vim known line/column, Emacs
  knows just the offset. The choice how to represent coordinates can
  be crucial. And of course there is that funny newline character,
  which is usually invisible in Vim except for empty lines. Because a
  motion is a important concept for many parts including operators and
  visual-mode. Probably a motion is represented by some abstract data
  type and this data-type is an important part of the interface a user
  must understand in order to write new motions and commands.
  Therefore it should be well-defined from the beginning because it
  cannot be changed easily afterwards.

These things should make the core and this has to be done right. Once
the core is ready most motions and commands are quite easy, probably
most can be copied from vim-mode or vimpulse, especially scrolling and
window commands, but also many others. If the definition of motions
and commands is clear, the implementation of those should be almost
independent from the core and I think this is very important. A single
command should just do what its purpose is and not interfere with the
rest. In fact, there are only two (or three) difficult commands: yank,
delete and paste, but I hope the implementation in vim-mode is quite
good know (it uses yank-handlers for all three kinds of yanking and
also provides "yank-pop" stuff). Note that these commands are
(almost?) independent from the rest of vim-mode, too.

Some parts of vim-mode are almost independent (or "orthogonal",
although I always have to smile when I read this word ;)) of the rest,
among them ex-mode, interactive search and the visual-stuff of
visual-mode. This should remain true for the new project.

> Whatever we decide, I think extensibility
> should be a prime goal.

I agree, whereas I would put the focus on new commands and motions.
The rest should be possible by defining clear interfaces. 

What could be useful are frameworks or perhaps only some functions
which help defining more involved motions like text-objects. We should
provide helper-functions for typical text-objects like parentheses,
blocks, whatever. vim-mode (and vimpulse, too) contains a few of those
functions which are again independent.

> In that regard, neither Vimpulse's nor vim-mode's keymap handling
> are quite as flexible as I'd like (I want to assign vi/Insert/Visual
> bindings to Emacs minor and major modes; this was a lot of work to
> accomplish under Viper). 

I read your comments and they sound reasonable. Adding auxiliary
keymaps should not be that difficult. As I remember correctly, local
keymaps were the biggest problem in vim-mode, but major and minor-mode
specific should be relatively easy. Of course this is very important
and should be done as one of the first things.

Note that vim-mode arranges keymaps in another hierarchy, too, which
allows to enable only few keybindings in certain major modes, e.g.,
only movement and window-commands in info-mode (somehow like
viper-on-more-modes I think). The difference is that many
standard-keybindings are not available in this case, e.g.,
insert-mode.

> My suggestion is to write an extensible core
> from scratch, and then "plug" the more mature parts into that
> framework.

Agree again. But we should try to cover as much aspects of the core as
possible from the beginning. Otherwise we will need to patch the core
later on as other functions are required and this can be problematic.
I hope that our experiences with vimpulse and vim-mode can help us.

> For what it's worth, I've pushed a sparse starting point to the
> "development" branch. The comments explain the keymap issue in
> more detail.
> 
> http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-states.el
> http://gitorious.org/vimpulse/vimpulse/blobs/development/vimpulse-tests.el
> 
> * Unit tests
> 
> I would really like to have a modicum of test coverage this time
> around. I don't how many Visual mode regressions I could have
> prevented if I had started writing tests earlier, but from my
> experience, some of the issues pertaining to Emacs' region
> (among other things) are naturally hairy.
> 
> Regarding test frameworks, I hear Christian Ohler's ert.el is now
> included with Emacs trunk, so we could go with that:
> 
> http://www.emacswiki.org/emacs/ErtTestLibrary

I never used any of those libraries but we should try. The main
problem is to cover all those funny special cases that arise (what
happens at the end of the buffer, on empty lines, ...) - most bugs
found in vim-mode are of this kind.

This is another point which is currently very bad in vim-mode. How to
deal with errors, should there be a unique signalling scheme, where to
handle them, ...

Btw, Emacs 23 contains visual-line-mode which is quite useful. But
most commands work on buffer lines not on screen lines. Should we try
to support visual-lines, too (optionally)? If yes this has to be
considered for motions and commands, especially for operator-stuff,
too, and we should think of it from the beginning, because many
commands have to be implemented differently for visual-lines.

Another thing somehow related to tests is documentation. We should not
only write developer-documentation but also user-documentation right
from the start. This should contain instructions for all important
concepts and also examples for new motions and commands of all kind.
Where should we put this documentation? EmacsWiki? Hosting site?
Perhaps as tex-info?

> 
> * Name?
> 
> Something new? Something catchy? Suggestions?

Hm, I think a new cool name would be good. I prefer short names or at
least names that allow short prefixes for function names ;)

I have no good idea so far, but I'll think about it ...

I really hope we can start a common project.

Frank