Split backend plugins into separate pip installable modules

[nuwang]

This issue is for discussing possible approaches to splitting out cloudlaunch plugin modules. It can probably only be taken up after this issue is addressed.

Plugin mechanism:

Our backend plugins currently all inherit from AppPlugin and are placed in a single plugins folder. It would be nice to replace this inheritance based approach with a more compositional approach and adopt a duck typed approach to plugin discovery. We can make each plugin a reusable django app that can be published to pip, each plugin capable of having its own set of endpoints etc.

Combined with a front-end plugin, we then only a need a yaml descriptor file that links the front-end and backend-plugin to create a final, installable artefact. We could probably store this yaml descriptor file along with the frontend module or in a git repo somewhere.

Plugin descriptor format:

The simplest format would be to mirror the current cloudlaunch admin setup

name: KubePlugin
version: 1.0.0
categories:
tags:
summary:
maintainer:
default_config:
description:
backend: <name_of_pip_package>
backend_component: <name_of_class>
front_end: <name_of_npm_module>
frontend_component: <name_of_class>

A more advanced format could include separate plugins for separate tasks. For example, one user interface for launching, and another user interface for configuring a running app through the "my appliances" page etc.

name: GVLPlugin
version: 5.0.0
...
...
interfaces:
       launch:
              - backend:
              - frontend:
       admin:
              - backend
              - frontend
       backup
              - backend
              - frontend

Publishing plugins:

Since plugins become a simple set of yaml descriptor files, installing them becomes a matter of providing cloudlaunch with a pointer to this file. We could have all descriptor files in a registry which are indexed and searchable through ElasticSearch, and exposed through a ReST API as an AppStore. We could also consider storing the plugin backend and front-end code in the AppStore itself for increased security.

Plugin installation:

Installing a plugin provides some challenges. One approach would be to simply pip install the backend plugin, and use django's built in capabilities to import the app at runtime and integrate it with the backend. and place the compiled front-end plugin in a shared web location somewhere. However, it should also be possible to directly load an app through a folder without having to go through pip, especially during development. The same dynamic import mechanism could probably be used.

However, it's less clear whether we can have multiple versions of a plugin running at the same time with such an approach or update a running plugin. This is discussed more below.

Plugin security:

Since we allow the execution of arbitrary code, our initial security model will have to rely on trust and manual vetting. In future, we can secure backend plugins by executing them within docker containers or similar, with configurable limits on resource access. The cloudlaunch backend can communicate with the plugin over an http shim. Each plugin could also be provided with an auto-generated, revokable API key, should it need to communicate with the cloudlaunch API for some reason. A dockerized approach would also solve the multiple version problem, since we can have multiple plugin versions running in separate docker containers.

For front-end components, since we allow the execution of arbitrary javascript, we would probably need to have a simple binary security model: trusted/untrusted. Trusted frontend plugins can execute arbitrary code. However, if a user decides not to trust the plugin the plugin must also have its UI defined in a yaml descriptor file using a restricted set of UI elements, similar to how galaxy tools, juju charms or openstack heat UIs are defined.