# Performance and benchmarking

### Overview

We have performed extensive automated load testing with our platform to measure performance, stability, scalability, availability, and cost with various types of loads. Our automated load testing tool *flowTester* is available publicly on [GitHub](https://github.com/cxfabric/cxf-benchmarking). The load testing was performed on our single instance (small) deployment of CXFabric, which is optimized for development and testing, not for performance, stability, scalability, or availability. 

{% hint style="info" %}
Quick look overview numbers of our benchmark testing.

* **1,000** Parallel flow executions
* **6.2** Second average runtime
* **16,000** Flow executions in 100 Seconds
* **100%** Success rate
  {% endhint %}

#### Setup

The test deployment consists of:

**Two** <mark style="color:blue;">t3a.large AWS EC2</mark> instances inside AWS EKS. Load testing took place with a “warm start” of the deployed platform, i.e., the platform had already been operating under some load prior to commencement of the load testing but the platform was not used for other purposes during load testing. Cloud cost examples can be viewed [here](https://cxfabric.gitbook.io/docs/scaling-and-performance/cost-examples).

Our load tests were parameterized, using different types of flows and different concurrency rates (parallel flow execution). Our latest load tests performed 25,000 executions of configured flows with concurrency rates between 25 at the low end and 1000 at the high end. In these tests, the *flowTester* triggered the execution of configured flows over the public Internet from a consumer-grade workstation.

We tracked the following metrics in our load testing:

* Number of successfully executed flows in an average 100 or 60 second time window
* Average flow execution time (ms)
* Percentage of successfully executed flows within 100 or 60 seconds of triggering execution
* Maximum CPU consumption (%)
* Maximum memory consumption (%)

#### Load Testing with 10 Different Flows, Each with 35 Nodes, with a REST Trigger <a href="#load-testing-with-10-different-flows-each-with-35-nodes-with-a-rest-trigger" id="load-testing-with-10-different-flows-each-with-35-nodes-with-a-rest-trigger"></a>

<table data-header-hidden data-full-width="false"><thead><tr><th></th><th></th><th></th><th></th></tr></thead><tbody><tr><td><strong>#concurrent flows</strong></td><td><strong>avg flow execution time (ms)</strong></td><td><strong>%flows successfully executed</strong></td><td><strong>#flows executed on average in 100 second time window</strong></td></tr><tr><td>1000</td><td>6218</td><td>100</td><td><strong>16082.34159</strong></td></tr><tr><td>750</td><td>5733</td><td>100</td><td><strong>13082.15594</strong></td></tr><tr><td>500</td><td>2418</td><td>100</td><td><strong>20678.24648</strong></td></tr><tr><td>250</td><td>1325</td><td>100</td><td><strong>18867.92453</strong></td></tr><tr><td>125</td><td>643</td><td>100</td><td><strong>19440.12442</strong></td></tr><tr><td>100</td><td>493</td><td>100</td><td><strong>20283.97566</strong></td></tr><tr><td>75</td><td>440</td><td>100</td><td><strong>17045.45455</strong></td></tr><tr><td>50</td><td>259</td><td>100</td><td><strong>19305.01931</strong></td></tr><tr><td>25</td><td>211</td><td>100</td><td><strong>11848.34123</strong></td></tr></tbody></table>

### Interpretation <a href="#interpretation" id="interpretation"></a>

We observe an ability of the platform to sustain high throughput, measured in number of flow executions in a given time interval (100 seconds or 60 seconds), for complex as well as simple flows. For simple flows (3 nodes), the platform executes more than 1500 flows in 100 seconds when a high number of concurrent requests is launched (>= 50). For complex flows (35 nodes), the platform executes more than 500 flows in a 60 second time interval when a high number of concurrent requests is launched (>= 50).

At the same time, we also observe high platform stability, measured as the percentage of successfully executed flows, despite the load tests running in a development deployment that is not tuned for maximum performance and stability.

***

### Cross Platform Comparison <a href="#cross-platform-comparison" id="cross-platform-comparison"></a>

CXFabric is quite similar to other iPaaS platforms such as n8n, a commercially successful iPaaS platform. n8n provides benchmarking information that makes a comparison useful. On its [benchmarking webpage](https://docs.n8n.io/hosting/scaling/performance-benchmarking/), n8n states “n8n can handle up to 220 workflow executions per second on a single instance.” To demonstrate this ability, n8n provides, as a graph, data from benchmark testing using a very simple two node flow on a “basic” single instance (ECS c5a.large instance with 4GB RAM). Architecturally, n8n uses a queuing model that queues any requests when all 220 “servers” are occupied. It then works off the queue in FIFO order as workflows complete and servers become available.

The CXFabric benchmark testing described above appears to be somewhat similar to the n8n benchmarking. A burst of requests is launched, all requests are allowed to process and complete, the number of successful completions within a given time window is noted, and another burst of requests is launched. This pattern of batches of requests is repeated to ensure that the platform continues to perform at the same level each batch and doesn’t show progressive degradation. To compare “concurrent” executions, the comparable tests are those that use a burst of 220 or more requests. Since there is no queuing in the CXF platform, comparing the “failure” rate in the CXF tests is instructive but not exactly comparable. However, the best direct comparison is shown below:

<table data-full-width="false"><thead><tr><th>Configuration</th><th>n8n</th><th>CXFabric</th></tr></thead><tbody><tr><td> </td><td></td><td></td></tr><tr><td>Basic Configuration</td><td>ECS c5a.large w/4GB</td><td>2*(ECS t3a.large w/ 8GB)</td></tr><tr><td>Workflow</td><td>2 steps</td><td>3 steps</td></tr><tr><td>Concurrent Flows</td><td>220</td><td>1000</td></tr><tr><td>Execution Time</td><td>Within 100 seconds (queued)</td><td>Real-time average 6 seconds</td></tr></tbody></table>