== Performance ==

Merlin put a close attention on performance.  All was done to provide answer
below reaction time for common queries.

The idea was that the work done for answering should be proportional to the
diff since the last query rather than the whole buffer size.

To this end, the design was a bit more complex than necessary and didn't evolve
well as new features were integrated.

A recent requirement put the final nail in the coffin for this principle:
syntax extensions expect to see the whole file at once, effectively defeating
the purpose of sub-linear optimizations.

This is the opportunity to redesign Merlin, in the hope of simplifying the work
for all components involved in the toolchain: the core of Merlin, the OCaml
typechecker, editors integration.

== Protocol ==

A new protocol will be designed. For compatibility reason, the old one will
still be provided on top, but the expectation is that editor modes will be
migrated to the new one over time.

The main idea behind the new protocol is that queries should be self-contained:
no implicit state is assumed between queries.

Executing a query on an existing process and on a fresh one should lead to the
same result, performance aside.

Intended benefits:

- reproducibility and stability; restarting always lead to the same
  behavior, otherwise it is a bug

- clear separation of responsibility; in particular buffer local state is
  managed by the editor, and not split between processes

- implementation is pure and performance comes from a memoization layer; the
  pure layer offers a reference implementation, memoization shouldn't be
  observable.

== Asynchronicity: editors requirements ==

NOTE: some of this might not be true in all circumstances, but would result in
twisted implementation or editor specific design.

*Vim* cannot do asynchronous operations.  *Emacs* cannot most of the time.
Otherwise it can be implemented in CPS-style but we have to be very careful.
*Neovim* and *Atom* should have no problem dealing with that.

As a consequence protocol is synchronous by default (Merlin will anyway process
queries serially) but editors can tweak the integration to provide better
experience.

== Asynchronicity: for optimization ==

A different kind of optimization can be explored later: refining results
asynchronously.

The idea is that for completion, changes that happened in the last second might
not be relevant for the query. Merlin could give an immediate answer from the
out-of-date cache, and refine it after recomputation.

The user always get a result in real-time, and potentially better suggestions
after a reasonable latency (the normal one).

The drawback is that this introduces more complexity on editor side. Not all
editor can support that kind of interactions:
- vim is out of scope,
- emacs will be hard to get right,
- modern editors such as Neovim and Atom should be easy,
- not sure about Sublime Text.

== Debugging ==

In the current version of Merlin, debugging can be done in two ways:
- via MERLIN_LOG file, where the whole communication is logged & some feature
  specific Printf-debugging is available
- via `dump` command, which exposes some internal structures.

The stateless protocol should help debugging and reporting bug:
- only the last command needs to be reported,
- users can first check the output in a fresh process.

The printf-debugging is feature specific and hardly readable.  The new
intention is to produce a human-readable trace at the same time the code is
executed, explaining intermediate decisions -- a dynamic counterpart to
literate programming.

As such, all internal structures will come with a human-friendly printer.

The trace should help *profiling*, by tracking times between steps.  The trace
should allow working at different level of precision, by unfolding sub parts of
the computation.  At the basic level, no internal decisions are printed and so
the cost should be negligible.  At the most verbose level, all steps are
printed, which is probably expensive but shouldn't matter in debug mode.

FIXME:
- explore using sturgeon for interactive tracing
- should we use ppx for deriving printers and logging code?

== Documentation & testing ==

Both are hard to achieve after the facts.  Each time a feature is ported to the
new implementation, documentation and testing should be added.

Documentation should not necessarily target the end-user but should explain
design rationale and intended use cases, so that end-user documentation is easy
to derive.

I don't know how to do proper tests... When a feature seems hard to properly
test, this should at least be documented, e.g. in a TODO file.  Otherwise,
tests should embed as most state as possible and not rely on external files
(OCaml / Opam setup, findlib packages, etc...).

=== Editor integration ===

Even more important is to do this when adding a feature in a specific mode.

Most users won't care about actual implementation details but will care about
how the feature is made accessible in their editor (prototypical example: local
type-enclosing keymap with C-up C-down C-w in emacs).

=== Documentation medium ===

Another question worth asking is how the documentation is provided and written.

For individual files, I am comfortable with markdown. Other text-based file
formats are welcome too if they prove more appropriate.

For multiple or structures files, I am considering directly using wiki
language, or markdown-like wiki.

== OCaml support ==

In current Merlin, OCaml frontend received a non-negligible quantity of
patches.

Recent changes in the OCaml compiler made integration easier, mainly
attributes/extensions and lazy substitution. The switch to PPX made support for
CamlP4 less relevant, most built-in extensions will be removed.

Merlin will still use a patched version of the compiler, but will try to stay
close to upstream.

Also some changes should be upstreamed soon:
- custom short-path implementation
- bidirectional typechecking of arguments
- state isolation
