Adobe Commerce & Queue Systems – integration review

Basis for Queue implementations on MMQF is  

\Magento\Framework\MessageQueue\QueueInterface. 

Both default Magento implementations (DB and RabbitMQ) are implementing this interface.

Let’s take a deeper look on it based on a flow.

This flow represents current Queue implementation, and as we see each service utilises its own functionality, but both of them are using the Queue Interface:

The table below describes all the methods expected to be implemented by the specific Queue provider and the intention of what functionality is expected from these methods.

In the following it is shown that each implementation realizes its own logic, which depends on the queuing system used:

• DB uses the use of its own tables in the database
• RabbitMQ implements the connection to the service and performs the required actions

On the other hand, we have consumers that fetch the message from the queue and process it according to the configuration.

Let's take a look at what other queuing technologies exist and if we can use them in Magento in the near future. Within messaging technologies, we can also divide them into the following categories based on different high-level use cases.

_{*Available with some limitations
**Other Aspects to consider (consumer groups, batching, multi-tenant seggregation, message encryption/security, aggregation, counting, scheduling, dead-letter queue)}

Since the basis for all queueing implementations in Magento is MMQF and QueueInterface, we will try to take a look at whether existing technologies implement the existing interface and can be easily integrated into the existing Magento architecture.

AWS EventBridge is a serverless event bus that facilitates the receipt of data from your application and from third-party providers to AWS Services. Currently, it appears that the destinations are specifically AWS Services. These destinations are defined via specific rules. 

More information can be found here: https://docs.aws.amazon.com/eventbridge/index.html 

And more about available targets can be found here: https://docs.aws.amazon.com/eventbridge/latest/APIReference/API_PutTargets.html  

But let's see if we can leverage AWS integration with the MMQF framework:

AWS MQ is a message broker based on the popular Apache ActiveMQ. It supports multiple protocols for connectivity, for example, AMQP, JMS, STOMP, NMS, MQTT, and WebSocket.

Most of the features are available because Magento also uses the AMQP protocol with RabbitMQ, but the protocol version is different, RabbitMQ uses 0.9 and Amazon MQ uses AMQP 1.0; therefore, any migration would require porting the queues from RabbitMQ to AWS MQ.

You can read more about this here: https://aws.amazon.com/amazon-mq/

In this case, everything looks much better. The reason is again that MMQF already uses AMQP

AWS SQS is a distributed and fault-tolerant queuing technology that provides point-to-point connectivity. It can be used with SNS to add a publishing and subscription mechanism. A single message is replicated on multiple SQS servers.

More information: https://aws.amazon.com/sqs/

SQS, as we can see, can also be the next queuing system in Magento, although it requires some effort to be implemented in MMQF

WS Kinesis is a streaming-based distributed messaging technology; it uses the publish/subscribe mechanism for loose coupling between senders and receivers. It is designed for extremely high throughput for real-time applications.

Streaming and the concept of stream itself are the central idea behind Kinesis. It is very similar to Apache Kafka, with some differences. Kinesis is also suitable for implementing event sourcing and CQRS patterns, which are often used in microservices architectures, as it provides immediate support for high-throughput messaging and publish/subscribe mechanisms.

Read more: https://aws.amazon.com/kinesis/

Kinesis also looks quite optimistic:

Apache Kafka is a widely used open source platform for stream processing and message passing. It is decentralized, replicated, and resilient (or fault-tolerant) and can achieve very high throughput.

The topic and publish/subscribe mechanism is the core of Kafka. Effective for implementing event sourcing and CQRS patterns commonly used in microservices architectures. Kafka is also used for a variety of streaming use cases that require near real-time processing of data sets.

More information: https://kafka.apache.org/

Kafka seems to be the more optimistic and valuable solution at the moment:

Microsoft Azure Service Bus is a fully managed message broker for enterprise integration. It supports familiar concepts such as queues, topics, rules/filters, and more.

Azure Service Bus supports AMQP 1.0 and some languages, PHP support is again limited for the protocol.

Main page: https://azure.microsoft.com/en-us/services/service-bus/#overview 

Since Azure also uses the AMQP protocol, it also looks quite optimistic. And the fact that Magento Cloud will also use Microsoft Azure makes the use of ASB even more likely.

Adobe I/O is a serverless, event-driven platform that lets you quickly deploy custom functions/code to the cloud without server setup. These functions are executed through HTTP requests or Adobe I/O events. These events can be orchestrated with sequences and compositions. The solution is based on the Apache OpenWhisk framework.

Events are triggered by event providers within Adobe Services, for example, Creative Cloud Assets, Adobe Experience Manager, and Adobe Analytics. To receive events for your application, you must register a webhook (URL endpoint) that specifies which event types it wants to receive from which event providers; Adobe sends events to your webhook via HTTP POST messages.

Home page: https://www.adobe.io/

In the current Magento implementation, it is not possible to use Adobe I/O. Therefore, an entirely new service needs to be developed to support it.

If we compare all possible solutions, the following picture emerges:

 _{*Workaround - The function may be available, but full support for PHP (library) is not available.}

As we can see, Apache Kafka is currently the winner of this research. Kafka is a very popular system that is performant and covers all the requirements we have with the growing complexity of Magento. Also in recent months, Kafka is one of the most discussed technologies as the next service Magento could use for asynchronous communication

However, integrators and developers can definitely look towards AWS MQ, AWS SQS, AWS Kinesis and Azure Service Bus integrations, as integrating these services does not require too much effort.

For these systems, one of the main problems is that AMQP 1.0 is not fully supported by PHP, but there are PHP Enqueue and Symfony libraries that provide an abstraction layer across multiple brokers.

In any case, as Magento service isolation approach insists on modules independence and extensibility, we must always be careful in the future direction of Magento asynchronous communication development. Because interfaces and architecture we build/extend now must be easily reusable and extensible in the future.

Photo by Levi Jones on Unsplash

CO-AUTHOR: