This part of the guidebook provides the details of the MELODIC architecture and explains how the platform can be built, installed or extended. It targets advanced users that would like to set-up, configure or even improve the Melodic platform.

1a. MELODIC Concepts and Architecture (SRL)

1c. Extending MELODIC / Plug in your mechanisms

This part of the guide offers an introduction to the MELODIC scope and provides valuable information about how multicloud applications can be modeled and managed through the MELODIC platform. It mainly targets adopters from the industry.

2a.Introduction/Scope of the Platform

MELODIC offers cloud agnostic and optimized management of multicloud applications. It is an open-source platform developed in terms of the H2020 MELODIC project. The MELODIC platform enables and optimizes data-intensive applications to run within defined cost, security and performance boundaries seamlessly on geographically distributed and federated cloud infrastructures.  The Melodic platform has a modular architecture with three main component groups as described below.

The Upperware: The Upperware can be termed as the brain of the Melodic platform. It has components that deal with finding optimal solution for the cross-cloud application and data placement problem, according to the user-defined requirements and in consideration of the current workload situation and involved costs.

The Executionware: The Executionware enables orchestration and monitoring of cross-cloud resources by providing abstract interfaces to various cloud platform APIs and tools.

The Modelling and Interfacing Tools: The modelling and interfacing tools include tools for data modelling, application modelling, and user-platform interactions. All user requirements are captured using a domain-specific language, CAMEL, which encapsulates all relevant aspects required for modelling data-intensive applications and their configurations in heterogeneous cross-cloud environments. The user-platform interaction is supported both in terms of platform APIs and web-based UI.

At the heart of the Melodic platform lies a feedback-driven adaptation loop that ensures that application deployments are continuously monitored, analyzed, and reconfigured, if needed, to ensure that the user-defined constraints and requirements are optimally satisfied for a given cross-cloud application. The interaction of the Melodic modules to realize this adaptation loop is shown in the figure below.

To adopt the Melodic platform for your multicloud application needs, you need to capture the details of your application (section 2b) and its respective constraints and optimisation goals using the CAMEL language. Next, you need to follow a number of simple steps for starting the initial deployment and management of your application by MELODIC. Along with these concrete instructions you may find an illustrative walkthrough that highlights the use of MELODIC in a real use case demonstrator.

2b. MELODIC Modelling

Melodic’s Metadata Schema provides a vocabulary though a hierarchical structure of all the concepts (represented as lexical terms) that are relevant for describing cloud application requirements, big data aspects and characteristics (with respect to the input and output of these applications) and the offered cloud infrastructure capabilities for discovering optimised multi-clouds placement opportunities. In addition, it encapsulates all the necessary concepts for enabling the context-aware authorization functions that the Melodic platform supports. The Metadata Schema comprises the Application Placement, Big Data and Context Aware Security models that group a number of classes and properties to be used for defining where a certain big data application should be placed; what are the unique characteristics of the data artefacts that needs to be processed; and what are the contextual aspects that may be used for restricting the access to the sensitive data. For example, if we need to know the geographical location where processing of a certain big-data type will take place, then a relevant geographical processing location concept should be defined, along with its related instances (e.g., EU, GR, ASIA) and properties (e.g., latitude, longitude). This Schema is used for formally extending the CAMEL language if need. Concepts from this Schema potentially affect the Requirement, Metric, Scalability, Location, Provider and Security sub-models of CAMEL. A bird’s eye view of the schema can be found in the following figure, while a detailed mind map for an easier walkthrough of the Schema’s main aspects can be found here: http://melodic.cloud/assets/images/MELODIC_Model_vFinal.png Also, a detailed explanation of all the classes and properties included in this Schema is provided here: https://melodic.cloud/wp-content/uploads/2019/01/D2.4-Metadata-schema.pdf

It is expected that the Metadata Schema will be adapted to the specific business and technical requirements of each application of a single organisation. For this purpose, MUSE – a Metadata Schema editor has been developed for creating, updating and maintaining the Metadata Schema. Typically, this task is undertaken by the administrator. Below we show how MUSE can be used.

Initialization

Before using the editor, it is necessary to fetch the Metadata Schema from the Models Repository into the Local datastore. This is achieved by clicking on the “Models repos. → Local repos.” menu item. The menu appears by clicking the hamburger style glyph at the upper left corner of the page as seen in the next Figure that illustrates the editor menu.

From the menu, the user can also transfer any changes made in the Metadata Schema, from the Local Repository into the Models Repository (menu item “Local repos. → Models repos.”) thus making them visible to the rest of the Melodic components. Other options include emptying the Local Repository (menu item “Clear Local repos.”), exporting Metadata Schema from Models repository (in XMI format), and reversely importing XMI files into the Models Repository.

Updating/Instantiating the Schema

By selecting “Metadata Schema management” from the menu, the Metadata Schema editor web page is loaded (as seen in the next figure). Using the tree-view in the left-hand side of the page, the user can navigate through the Metadata Schema. Concepts are represented with yellow folder icons in the tree view, whereas other artifacts (e.g. concept properties) are represented with coloured balls. By selecting a tree node, its details are loaded into the “Node Properties” form, found in the main area of the page. Furthermore, the form fields change according to the type of the tree node selected (i.e. concept, concept property or concept instance). For example, the Range form field is only available for the concept property nodes. The white text fields can be edited while the grey ones are read-only. At the bottom of the page, there is a row of buttons for creating new nodes (concepts, properties, and instances), deleting the selected node or saving changes in the form.

In the remaining section, a simple use case, where a new sub-concept will be created, is detailed. First, the user selects the “Data management” node in the tree view, as shown in the Figure below. This node will be the (immediate) parent concept of the concept to be added. Upon selection, the parent node’s data are fetched from the Local Repository and are displayed in the “Node Properties” form. Next, the “Create Concept” button must be pressed to start creating the new sub-concept.

The same functionality is also available through a context popup menu, which is available by right clicking on the parent node, and then clicking the “New Concept” item (as seen in the next Figure).

Both actions (i.e. pressing “Create Concept” or clicking “New Concept” in context popup menu) will make the “Node Properties” form (in the main page area) to adapt in order to include only those fields that are relevant to Concepts (i.e. Id, Parent Id, URI, Type, Name, Description). The values of Id and URI fields are automatically completed, as shown in next Figure.

The user can modify Id and URI values if needed, fill-in the new concept’s name, for instance “Data Visualization”, and optionally give a description, in the corresponding fields. Eventually, by pressing the “Save Changes” button, the new sub-concept’s data are submitted to the Local Repository for storing. Afterwards, the tree view is refreshed in order to include the newly added concept.

2b-3. How to model your application using CAMEL

CAMEL is a multi-domain-specific language (multi-DSL) which is specialised in the modelling of multi-cloud applications. It comprises multiple domains which cover information aspects relevant to the multi-cloud application lifecycle. These aspects include deployment, requirement, metric and scalability. CAMEL currently advances the state-of-the-art due to its richness and deepness in the way the various aspects are covered.

CAMEL is accompanied with an editor which can assist in the development and modification of CAMEL models. Specifically, an Eclipse-based textual editor is offered which enables the editing of models that conform to the textual syntax of CAMEL. In this respect, instructions on how to use the CAMEL’s textual editor can be found here: [CAMEL] Camel 2.5 Eclipse (oxygen) editor installation.

As indicated above, CAMEL captures multiple aspects. In fact, a CAMEL model of a user application can be seen as a container of sub-models, each of which maps to one of these aspects. Please note that not all aspects can be edited by users. In particular, the execution aspect is meant to be touched and maintained only by the MELODIC platform. Further, for some aspects, the type-instance pattern is followed. In that case, the user should specify only the relevant aspect-specific models at the type level as the instance level is again maintained by the MELODIC platform (with the sole exception of the data aspect where the user can specify concrete data and data sources (instances) which pre-exist before the execution of his/her application). In this respect, we supply in the next table the following information per each aspect covered by CAMEL: (a) the actual aspect; (b) key concepts from this aspect; (c) a link dedicated to explicating how models pertaining to this aspect can be specified.

2c-1. Initial Deployment

1. Make sure that your MELODIC instance is ready for the deployment: it is freshly restarted and all components are ready. Please look at Preparation of the Melodic machine after IP change.
2. Go to https://{{MELODIC_IP}}
3. Log in using your LDAP’s credentials
4. Download the chosen CAMEL Model file. The basic example is available here: https://s3-eu-west-1.amazonaws.com/melodic.testing.data/TwoComponentApp.xmi
5. Go to the new deployment bookmark
6. 
7. Drag and drop downloaded CAMEL (.xmi) file and upload it to the CDO Repository
8. 
9. When the CAMEL Model is uploaded, go to the next step.
10. 
11. Choose which application you want to deploy and which cloud providers you want to use.
12. 
13. Go to the next step and push the green button.
14. In a minute, you will be moved to the deployment view, where you can observe the deployment process. If everything goes correctly, you should have the view similar to this: 
15. If you want to have a more detailed view, it is possible to see the process view using Camunda on {{MELODIC-IP}}:8095 (login with your LDAP’s credentials)
16. Please take a look if your instances have opened the necessary ports. For this basic example it is 3306 on the database and 3306 and 9999 on the application instance.
17. When your application is deployed, you can check the result in the Your Application bookmark. 
18. Let’s do a simple test if application works properly:  http://{{application-ip-host}}:9999/demo/all
19. Save name and e-mail to the database http://{{application-ip-host}}:9999/demo/add?name=Name&email=email@melodic.com
20. Check if it has been saved. http://{{application-ip-host}}:9999/demo/all
21. Application works as expected.

Preparation of the Melodic machine after IP change

1. If you are not connected with the Melodic machine, open a terminal and login

   ssh -i {{nameOfYourKey}} ubuntu@{{MELODIC-IP}}

2. If your instance has a new IP, use command:

   ipupdate

3. Restart the Melodic using command:

   drestart

4. Wait until or check if all components are ready. You can check the status of each component using command:

   mping

5. If your Melodic instance has a new IP, add GUI-backend self-signed certificate in your browser. Open https://{{MELODIC_IP}}:8078 and confirm the security exception.

   

6. After that, you will see:
7. 
8. Open MELODIC GUI: https://{{MELODIC_IP}}
9. If your Melodic instance has a new IP, add GUI-frontend self-signed certificate in your browser by confirmation the security exception.
10. Now your Melodic instance is ready for the deployment.

2c-2. Reconfiguration Management

Definition – What is reconfiguration

Reconfiguration is the process of modifying a multi-cloud application’s (current) deployment into a new one that better addresses application’s constraints and objectives, as they are captured in its CAMEL model

Initial Deployment vs. Reconfiguration

CAMEL Model elements that affect reconfiguration

• Requirements model:
  • Optimisation Requirements: require maximization/minimization of certain metric variables
  • Service Level Objectives (SLOs): they are related to constraints that require taking action when they are violated
• Constraints model:
  • Metric Variable constraints: will require updating CP model with current metric variable values
• Metric model:
  • Metric Variables: will require updating CP model with their current metric variable values

Metric model metrics also specify where and how application and Virtual Machine (VM) monitoring info is gathered (each type of metric is assign to a specific Monitoring hierarchy level, i.e. VM/Host, Cloud Zone/Region/Provider, Global)

Steps of reconfiguration

• Measurements are collected by sensors deployed in application VMs and components
• Measurements are sent to local EMS agent (called EMS client)
• EMS agent applies event processing rules (each measurement is an event) for Host/VM monitoring level
• EMS agent forwards events needed to higher monitoring levels to the appropriate EMS agent
• Intermediary EMS agent (if any) receive events from lower level EMS agents
• Intermediary EMS agent applies event processing rules for Cloud Zone/Region/Provider level
• Intermediary EMS agent forwards events needed to higher monitoring levels to the appropriate EMS agent
• EMS server (part of Melodic stack) receives events from lower level EMS agents
• EMS server applies event processing rules for Global monitoring level
• MetaSolver receives certain Global EMS agent event processing results:
  • some are used to update application CP model with current metric values (can be composite, e.g. average execution time, max RAM usage)
  • some are used to indicate that a constraint has been violated or a condition calling for application reconfiguration has occurred
• MetaSolver signals the Melodic stack’s Control process to start a new reconfiguration iteration
• Control process starts the Solving process (which solves a mathematical constraints problem to come up with a suitable application deployment)
  • A CP problem solver is selected
  • Solver solves the CP problem using the current metric values (those updated by MetaSolver with EMS info)
  • New solution (corresponds to a new application deployment) is checked if its significantly better than current one (thus preventing pointless redeployments)
  • Adapter finds the differences between current and new solution and realizes the necessary application VM deployments and undeployments
  • In new deployments, apart from application components, sensors and EMS agent are also installed
• New VM EMS agents connect to EMS server and get their configuration

3-1. Introduction to CAS demonstrator

Environment

CAS Software AG

Founded in 1986, CAS Software became the german market leader for xRM software for small to mid size enterprises, a generic software allowing not only customer relationship management but any relationship management. CAS has a wide partner network, distributing and adapting the xRM software to customer and business sector needs. As especially small enterprises lack the infrastructure and experts for hosting the xRM system, CAS also developed a cloud solution for xRM called ‘SmartWe’. As the on-premise software, SmartWe allows customer-specific data types, forms and workflows to be developed in form of apps being part of the web application itself.

SmartWe

SmartWe (Link) is not only the software system itself but stands also for an ecosystem and a form of cooperative: the SmartWe World AG. Members of SmartWe World AG contribute to the ecosystem e.g. through marketing or development of apps. As customer needs in diverse business sectors are hard to grasp and generalize, SmartWe is developed as a generic platform with a sophisticated SDK (software development toolkit). With the SDK, development partners in the SmartWe World ecosystem are able to customize existing and develop new apps targeted to specific business sectors and customer needs.

The SmartWe platform and basic xRM solution is hosted in german datacenters, ensuring conformance to german data protection regulations. It consists of a set of standard xRM apps, a sophisticated search service, the SDK and an app store. Via the app store, customers can install additional apps to their web based SmartWe solution. With the help of Melodic, it´s possible to deploy different services on different infrastructures yet having on central monitoring and optimization. For example, the database can be hosted in an own managed infrastructure, whereas the client side can be deployed at Amazon in Germany and app providers can even bring their own nodes to let their application be hosted on a self-managed infrastructure, if desired. Each service´s requirements are described in one central Melodic model, having dedicated optimization and scalability rules allowing high performance while considering costs. By monitoring real time sensors, the services and data access are scaled up and down according to the actual usage of the component.

Demonstrator

The demonstrator of CAS Software AG bases on their CRM application ‘SmartWe’ with the following software components:

1. HAProxy (Link) load balancer
2. ‘SmartDesign’ frontend
3. OPEN backend server
4. MariaDB (Link)
5. External application(s)

The CRM application with its components shall be automatically deployed with the MELODIC platform. All necessary steps from the platform setup to the actual application deployment and automatic optimizations are described in the following chapters.

Install Melodic

The installation of Melodic is performed on a virtual machine managed by the IT department and configured according to company standards (network, backup, administration access polcies, ..). Both the Melodic upperware and the executionware ‘Cloudiator’ (Link) are installed on the same host, what requires slightly more hardware resources as if both were installed on separate machines. The decision for a well-equipped machine also leaves room for future extensions. The Melodic machine at CAS Software AG has the following specifications:

Platform

Once the machine is ready, Melodic can be installed by checking out the project’s ‘util’ repository. It contains all the required installation scripts and allows – based on its branching model – the selection of a specific version. For the time being, R2.5 is the latest available release.

The script first prepares the environment by installing Docker (Link) and then copies the initial Melodic configuration and docker-compose.yml files. The installation script has an average total execution time of around 2 minutes on the machine type (see description above) used within the demonstrator.

Configuration

Melodic comes as almost ‘zero configuration’ platform that detects required configuration settings automatically and without the necessity for manual adjustments. The only exception is the initial user that has to be added individually. While an LDAP user interface (e.g., ) can be used, a simpler way is to use the command line tool that comes with Melodic. It guides through the creation process and prompts for the required information (e.g., name and password).

Once the LDAP user is added to the installed Melodic platform, a first platform start can be initiated. This can either be done for the upperware and the executionware separately by using the respective docker-compose.yml files, or by simply using the built-in commands Melodic is shipped with.

Verification

The installation can easily be verified by running the following built-in command:

The command performs a local TCP port scan against the services the Melodic platform exposes. It gives a first hint on whether a service is generally available or not. Checks on application level have to be done apart if needed. A good indicator for the platform being up and running is the Melodic UI since it contacts different components to gather information about the system’s overall health.

Monitoring

Melodic and its components are optionally monitored at CAS Software AG with the monitoring solution Check_MK (Link). It allows to monitor standard metrics such as the CPU load or the memory and can be extended by application-specific service checks. At CAS Software AG this was done for some Melodic components (e.g., UI frontend and backend, Grafana, Cloudiator REST API, message queue, ..). Once the state of a relevant Melodic component changes, a notification is sent out to the responsible DevOp.

Prepare Application

To allow Melodic the handling of an application, it first has to be modelled in CAMEL. Since every application comes with its own requirements and characteristics (e.g., installation, communication) it is recommended to start off with some conceptional work leading to the application model’s design. In case of SmartWe, the application components were first split up into separate components with clearly defined communication dependencies. The defined communication will later determine the deployment/initialization order of the application based on the dependencies that have to be resolved.

In a second step, meaningful requirements for each component are defined. These requirements stem from both experiments and observations from the conventional deployments before Melodic. Whether a public or private resource is required for a concrete component most certainly depends on company policies. In case of SmartWe, three deployment types were defined:

Since all components of SmartWe now base on Docker containers, their respective images have to be defined. Since all components use custom images, an internal Docker registry is used. The credentials are provided as component parameters.

Modelling

The modelling in CAMEL of the application itself is explained in detail for the frontend component ‘SmartDesign’. As a first step, a ‘software’ entity named ‘SmartDesign’ is added to the yet empty model. Each line is explained with a comment directly in the following codeblock.

In the next step, the component requirements referenced from the previously explained ‘software’ entity are defined. Each requirements represents another reference to their prameterized requirement type in the global requirement model. Their meaning is explained by comments in the codeblock below.

The global requirement model contains the exact definitions for each requirement. The below codeblock shows its content for the ‘SmartDesign’ frontend component explained with inline comments.

To fully support efficient reasoning, a utility function is defined that optimizes for cost (e.g., find the cheapest possible solution). The closer to ‘1’, the more efficient the solution is. For the ‘SmartWe’ demonstrator, the cost of a ‘SmartDesign’ frontend component multiplied by the cardinality is used as denominator.

A constraing representing the SLO that initiates an optimization is defined based on the average overall system RAM being always below 90%. Once the 90% are crossed towards higher values, the SLO is fired by the EMS causing Melodic to start another reasoning process in order to find a ‘better’ deployment situation. In case of the demonstrator this leads to adding another instance of the ‘SmartDesign’ frontend (due to considering the cardinality in the utility function). Generally it could also be an option to replace the existing deployments with one done on a bigger machine type.

Secure Operation and Deployment

After having the CAMEL application model ready, some additional security is added by creating XACML rules that are then added to the Melodic authorization mechanism. Based on some internal considerations, the following must be enforced before any deployment is made:

The codeblock below shows the XACML-based representation of the two described security rules. Over the whole policy, standard XACML is used that could be extended at any time if necessary. The file itself is stored in the ‘~/conf/policies’ folder and automatically loaded by the authorization mechanism during runtime.

Deploy Application

With the CAMEL application model and the security policies ready, the initial deployment itself can be performed. To ensure a running Melodic platform with all components up and ready, the ‘mping‘ command can be used a last time. After verifying that all components show the status ‘OK‘, the Melodic UI can be opened in the browser. The credentials of the initial LDAP user are used for the login.

Provider Settings

In case of the CAS Software AG demonstrator around ‘SmartWe’, two providers are configured where one of these represent a public and the other a private cloud resource. The UI menu ‘Provider Settings’ is used to navigate to the provider overview, where “+ Add” is used to configure a new provider.

In case of the Amazon AWS public cloud, settings are made accordingly. The provider preset ‘aws-ec2’ can directly be choosen from a dropdown list, since its connectors are already contained in Melodic. Other provider presets (e.g., Azure, GCP) are available as well. Crucial for the provider setting are the credentials. In case of AWS a ‘user’ and a ‘secret’ are provided in the respective input fields. The credentials are stored securely in a component named ‘secure variable store’.

The same procedure is repated for the OpenStack private cloud. Here, a specific endpoint has to be provided whereas the Amazon AWS configuration did not require one due to being a publicly accessible under a unified, single URL. The ‘user’ as part of the credentials is derived from an installation (default), a project identifier (d829a..) and a concrete user (cas). The ‘node group’ is set to ‘cas’ what allows for an easy identification once the component is deployed. With most providers, Melodic will use the node group as part of the instance name.

The existance of cloud providers can easily be checked on the ‘Provider Settings’ page of the Melodic UI.

Deployment

Once the providers are setup and ready to be used, the deployment itself can be initiated. Under the “New Deployment” menu on the Melodic UI the user will find a wizard that prompts for the required information. The steps are as follows:

Once the deployment is successfully done, the deployment can be inspected by navigating to the Melodic UI menu ‘Deployed Artifacts’. Besides technical details about the underlying Cloudiator ‘Jobs’, ‘Schedules’ and ‘Processes’ there is a view for the allocated nodes. It shows information about the application component instances and their IP addresses. A similar view is offered within the menu ‘Your Application’ with a stronger emphasis on direct interaction with the machines. For development purposes, the SSH connection feature can be used to directly connect to the machine’s host OS.

Apart of the built-in possibilities of checking deployments it is of course always practical (especially during the development tests) to visit the cloud providers management UIs. The screenshot below shows the AWS ‘EC2 Management Console’ with all public components of ‘SmartWe’ successfully deployed and running. Checking the management console is especially useful if provider-specific tools (e.g., billing details, detailled checks) are needed. If only basic information is required, the Melodic UI alone should be sufficient since it queries the cloud provider for some of its data.

Visualization

Metrics produced by the Melodic sensors and the EMS are a useful tool when it comes to monitoring the health status of the overall application and its components. Depending on the sensors defined in the CAMEL application model, the respective values can be analysed and used within dashboards with Grafana. The required environment is installed as part of the Melodic platform and can be accessed under port 3000. For the demonstator, a meaningful dashboard was designed based on the available sensor metrics (RAM usage) and generally available meta information (nodes, SLO variables like thresholds).

Optimization/Reconfiguration

Based on the modelled constraint (see above) Melodic initiates an optimization of an existing deployment whenever the SLO is fired. To test the rules and the underlying mechanism it turned out to be good practise to provoke related system states with the Linux tool ‘stress’. On the ‘SmartDesign’ instances RAM usage was forced to certain levels (>90%). A script was written for that purpose allowing to pass the targetted RAM usage as parameter. The target values are not very precise and should only be seen as an estimation.

As a result of the optimization, another instance of ‘SmartDesign’ is added and registered at the load balancer. The screenshot below shows the Grafana dashboard shortly after an optimization was done. The ‘Instance’ indicator increased from 5 to 6 and an SLO event is logged in the “SLO Scaling Event” graph. As for the initial deployment the already described ways of checking a deployment (e.g., ‘Your Application’ view, cloud provider management console) can be exploited to check the optimized deployment situation.

3-2. Walkthrough video