Once issue730 is merged, we want to extend tab support to also work in the --search component. To recognize the context from the incomplete string, we have to do some limited parsing. I suggest to handle this similar to argcomplete which runs parser.parse_known_args() to use the normal arg parser until it fails but keeps everything that was parsed until then. 

For us, this is a bit more complicated because our multi-stage parser for option strings will not produce a valid AST in the first stage, so the second stage that annotates the AST with features, will not run. I suggest to modify it, to introduce a method that does a partial parse with the assumption that after the last token, all currently open AST nodes are closed correctly. I'd try to add "Incomplete*" nodes to the AST to represent the partial part of the parse.

For example, I'd like to parse a string like 
  "astar(lmcut(transform=co"
first into tokens
  ["astar", "(", "lmcut", "(", "transform", "=", "co"]
and then into the AST
  IncompleteFunctionCallNode(
    name="astar",
    positional_arguments=[
      IncompleteFunctionCallNode(
        name="lmcut",
        positional_arguments=[],
        keyword_arguments={
          "transform": IncompleteTokenNode("co")
        })
  ])
and then finally, into the enriched AST:
  EnrichedIncompleteFunctionCallNode(
    feature=AStarFeature(),
    arguments={
      "eval": EnrichedIncompleteFunctionCallNode(
                feature=LMCutFeature(),
                arguments = {
                  "transform": IncompleteTokenNode("co")
               }
    })

From there, we could then make suggestions because we know that the word "co" is the start of the "transform" argument of an LM-cut feature, so we have to suggest task transformations starting with "co".


I think this would also work on other levels: for example, with the input "astar(lmcut(", we'd get
  IncompleteFunctionCallNode(
    name="astar",
    positional_arguments=[
      IncompleteFunctionCallNode(
        name="lmcut",
        positional_arguments=[],
        keyword_arguments={})
  ])
and could suggest
  * ")" to complete the innermost IncompleteFunctionCallNode
  * anything that works for the first positional argument of lmcut
  * "k=" for any k that is the name a parameter of lmcut.
The difference to the case above is that here the innermost IncompleteFunctionCallNode doesn't have any incomplete child nodes and no complete arguments so far, otherwise we should prepend our argument suggestions with ", ".


Not sure if this would works for all cases though. Any thoughts on the design?