This part of the guidebook provides the details of the MELODIC architecture and explains how the platform can be built, installed or extended. It targets advanced users that would like to set-up, configure or even improve the Melodic platform.
1a. MELODIC Concepts and Architecture (SRL)
The Upperware: The Upperware can be termed as the brain of the Melodic platform. It has components that deal with finding optimal solution for the cross-cloud application and data placement problem, according to the user-defined requirements and in consideration of the current workload situation and involved costs.
The Executionware: The Executionware enables orchestration and monitoring of cross-cloud resources by providing abstract interfaces to various cloud platform APIs and tools.
The Modelling and Interfacing Tools: The modelling and interfacing tools include tools for data modelling, application modelling, and user-platform interactions. All user requirements are captured using a domain-specific language, CAMEL, which encapsulates all relevant aspects required for modelling data-intensive applications and their configurations in heterogeneous Cross-Cloud environments. The user-platform interaction is supported both in terms of platform APIs and web-based UI.
Each of the Melodic modules is conceptually divided further into components. The Melodic components interact with each other through well-defined interfaces encapsulated over two separate integration layers: the Control Plane and the Monitoring Services. Melodic’s implementation of the Control Plane is based on an Enterprise Service Bus (ESB) architecture with process orchestration through Business Process Management (BPM). The ESB architecture utilizes a centralized bus for message propagation between components. Components publish messages to the ESB, which are then forwarded to all subscribing components. BPM orchestration is used to orchestrate invocation of methods from underlying Melodic components. The Monitoring Services involve a secure complex event processing framework for efficiently aggregating and processing highly distributed monitoring data.
A Feed-back driven Adaptation Loop: At the heart of the Melodic platform lies a feedback-driven adaptation loop that ensures that application deployments are continuously monitored, analyzed, and reconfigured, if needed, to ensure that the user-defined constraints and requirements are optimally satisfied for a given cross-cloud application. The interaction of the Melodic modules to realize this adaptation loop is shown in the figure below.
As depicted in the figure, the user requirements and constraints, related to both applications and data, are captured using CAMEL. The Upperware takes CAMEL model as an input and calculates the optimal data placements and application deployments on aggregated cross-cloud resources in accordance with the specified application and data models in CAMEL. The actual cloud deployments are carried out through the Executionware.
The Upperware: A high-level overview of the Upperware components is shown in the figure below. The Upperware workflow is involved in both finding an optimal initial placement topology for the cloud applications on the appropriates cloud resources, as well as keeping the deployed solution optimized based on the collected monitoring data.
The user-defined application and data model is given to the Upperware in the form of CAMEL. Then, a series of Upperware components are involved in finding an optimal deployment solution, as well as adapting the cloud resource deployments for an application. The process begins with generating a Constraint Programming (CP) model that is processed and solved by Melodic’s Solvers. As Melodic’s approach is based on the utility-based optimization, a dedicated Utility Generator is used to evaluate different candidate solutions that each solver may suggest. The utility is based on a user-defined utility function that also integrates the utility value calculated by the Data Lifecycle Management System (DLMS). The DLMS employs various data-aware algorithms that takes data aspects of the application deployments in consideration when calculating the utility. Finally, the Adapter component analyzes and validates the calculated deployment model, and defines the particular reconfiguration action tasks to be executed in a specific order by the Executionware in order to implement the optimized deployment configuration for the current execution context found by the solvers.
The Executionware: The main jobs of the Executionware are the management of diverse cloud resources in a provider-agnostic way, the orchestration of data intensive applications on top of these cloud resources, and the deployment of the monitoring services to collect run-time metrics. The Melodic Upperware interacts with the Executionware in order to deploy data intensive applications, specified in the user-supplied CAMEL application model, to the Cloud.
The core functionality of the Executionware is built upon the cloud orchestration tool, called Cloudiator. A high-level view on the Executionware components and its interaction is depicted in figure below. Besides Monitoring Services, which are integrated with the Monitoring Services of the Upperware, the Executionware also includes a holistic Resource Management Framework and a Data Processing Layer to enable orchestration and management of Big Data processing frameworks and user jobs on the cross-cloud resources.
The Melodic Interfaces to the End-Users: For modelling Cloud applications and data, the Melodic’s users use CAMEL language, which is a DSL developed across several cloud projects and extended in the Melodic. For modelling applications and data, two CAMEL editors are available, a textual CAMEL editor and a web-based CAMEL editor. The textual CAMEL editor is based on Eclipse IDE and allows Create-Read-Update-Delete (CRUD) operations over the main CAMEL elements for describing, among others, the decomposition of the Cloud application into its components and for defining placement and scalability requirements that follow the required service level objectives (SLOs). Moreover, data sets are also modeled via the same editor.
Melodic also supported the capability to extend CAMEL according to the requirements of the Melodic adopters, based on the Metadata Schema. The CAMEL metadata schema constitutes a formal way for defining extensions in CAMEL.
The Melodic platform also provides a web-based UI, which enables easy application deployments and monitoring.
1b. MELODIC Build and install
The section contains a detailed description of the MELODIC platform and additional tools installation.
MELODIC PLATFORM installation
Melodic installation procedure could be found here
CAMEL Textual Editor
As indicated in section 2-b3, CAMEL involves a textual editor that is currently promoted as it fits devops preferences. In this respect, we supply here guidelines on how to install CAMEL’s textual editor in an Eclipse Environment (mapping to the Eclipse oxygen version). This is a standalone installation irrespectively of the place where the MELODIC platform is installed.
This tutorial describes how to install the Eclipse oxygen-based Camel editor.
- install the Eclipse – oxygen version, for example “Eclipse IDE for JAVA Developers version” (follow instructions on: http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/oxygen/3a/)
- inside the Eclipse , choose help-> install new software
- choose Oxygen – http://download.eclipse.org/releases/oxygen
- choose the following plugins:
- EMF – Eclipse Modeling Framework SDK 2.13.0.v20170609-0928
- EMF Parsley CDO 1.2.3.v20180220-1246
- OCL Classic SDK: Ecore/UML Parsers,Evaluator,Edit 5.3.0.v20170607-1133
- install more plugins:
- OCL Examples and Editors SDK 6.3.0.v20170613-1432
- Xtext Complete SDK 2.12.0.v20170519-1412
- and more:
- CDO Model Repository SDK 4.6.0.v20170602-1611
- Download camel folder from Bitbucket (branch: rc2.5): https://bitbucket.7bulls.eu/projects/MEL/repos/camel/browse/camel?at=refs%2Fheads%2Frc2.5
- Import camel projects into the eclipse workspace
- generate mwe2 flow ( right click on GenerateCamelDsl.mwe2 under src/camel/dsl and then runas->MWE2 workflow)
Running the textual editor:
- right click on camel.dsl project and run as -> eclipse application
- create general project (just once) and put .camel files inside
- copy MMS.camel, location.camel, metric.camel & unit.camel files (from camel repository->examples) to the workspace (this enables to insert Metadata Schema annotations in the CAMEL model as well as re-use metric, unit & value type elements from the three latter template models)
- open camel file (for the first time it will ask to enable xtext nature – just agree and remember the decision)
- after editing and saving .camel file the .xmi file should be generated under src-gen folder
Once the first execution alternative is followed, you can open the Debug view (Window→Show View→Other→Debug→Debug).
There, in the opened view, you actually see the command that is being executed. You can just right click on it and select Properties.
This will unveil the actual command string. You can then just copy and paste it, e.g., in a certain executable file like run.bat for Windows.
Then, you can just stop the previous execution and use the respective executable file to launch the editor from now on without having to run a priori any (graphical) Eclipse Environment.
This can certainly enable to save a lot of main memory space.
Note: this command is workspace-specific so this is the reason that requires first to create the right workspace and have the right Eclipse plugins installed as well as the CAMEL ones.
1c. Extending MELODIC / Plug in your mechanisms
The MELODIC platform is an advanced autonomous middleware that acts as an automatic DevOps for cloud applications, able to make the appropriate adaptations to the deployment configuration as the application’s context changes.
In MELODIC the application details, its components, the data sets to be processed, and the available monitoring sensors can all be described in a Domain Specific Language (DSL). The application architecture description is coupled with the goals for the deployment, the given deployment constraints, and monitoring information to allow the MELODIC platform to continuously optimize the deployment. Therefore, MELODIC follows the models@runtime paradigm.
The MELODIC architecture has the following main layers:
- The application modelling based on the the Cloud Application Modelling and Execution Language (CAMEL);
- The Upperware solving the optimization problem and adapting the deployment model;
- The Executionware based on the Cloudiator cross-Cloud orchestration tool responsible for the deployment of the solution decided by the Upperware.
For integration purposes a professional ESB (Enterprise Service Bus) is used, based on MuleSoft ESB (community edition). For the business logic and service orchestration, the BPM solution based on Camunda BPI is used. Based on these architectural choices, it is quite straight forward to extend the MELODIC platform by adding new components and configuring the proper logic flow updating the BPM process.
The MELODIC architecture is presented in the figure below.
Integrating components from different development teams into MELODIC platform makes it mandatory to cater for high availability and reliability. It is important to ensure a reliable flow of the invoked operations, guarantying full control over an operation’s execution. It must, therefore, provide unified exception handling and retries of operations. This requires the ability to monitor all operations invoked on the integration layer, with a configurable level of detail, and configurable and easy usage of a single logging mechanism for all the invoked operations. The integration framework selected for MELODIC platform supports flexible orchestration by the ability to easily set up and reconfigure the orchestration of method invocations for any of the underlying components. It is possible to configure such orchestration without the need to code and recompile the whole platform every time an extension takes place. This implies that the MELODIC framework has support for the most commonly used integration protocols: Simple Object Access Protocol (SOAP), Representational State Transfer (REST) and the Java Messaging Service (JMS). It also supports both synchronous and asynchronous communication methods with an easy way to switch from one to the other. Also, it has the ability to perform complex data model transformations.