Installation

To use the process pipelines:

Composer must get installed
The associated modules are registered

Installation for Magento

composer require pacemaker/component-process-pipelines (1)

bin/magento setup:upgrade (2)

1	Install the Component Pacemaker Process Pipelines using Composer
2	Register the Modul in Magento

Installation Standalone

To install Pacemaker Process Pipelines standalone core functionality, only the Pacemaker Process Pipelines component is required.

composer require pacemaker/component-process-pipelines ^2.5 (1)

1	Install the Core Component Pacemaker Process Pipelines using Composer to run it standalone

Example

Composer JSON installation file

{
  "name": "pacemaker/component-process-pipelines",
  "type": "metapackage",
  "require": {
    "php": "~7.4.0||~8.0.0||~8.1.0",
    "techdivision/process-pipelines": "^2.5.0",
    "techdivision/pacemaker-base": "^2.0.1",
    "techdivision/pacemaker-pipelines": "^2.0.2",
    "techdivision/pacemaker-pipeline-initializer": "^3.3.0", (1)
    "techdivision/pipeline-step-processes": "^2.0.2",
    "techdivision/pipeline-step-messages": "^2.0.0",
    "techdivision/pacemaker-pipelines-monitor": "^2.2.0", (2)
  }
}

1	See Pacemaker Pipelines Initializer
2	See Pacemaker Pipeline Monitor

The version numbers of the individual modules here are only an example
They are bound to the respective component version please do not consider

Environment configuration

Configure Heartbeat Cron

The heartbeat cronjob is one of the significant parts of the Pacemaker. It is necessary to have this job up and running to use Pacemaker.

The heartbeat executes the condition checks for and initiates pipelines, which are ready to be completed. At the same time, it checks the conditions for steps within existing pipelines and triggers the execution if a step is ready to run.

Default configuration

By default you should create a new crontab entry within your OS as following:

* * * * *  /usr/bin/env php <PATH_TO_MAGENTO_ROOT>/bin/magento pipeline:heartbeat

This configuration will run the heartbeat every minute

Another heartbeat interval

If you need to run the heartbeat less often than every minute, you must define the pulse to avoid time gaps within time-based conditions.

Example for a heartbeat, which is running every two minutes:

*/2 * * * *  /usr/bin/env php <PATH_TO_MAGENTO_ROOT>/bin/magento pipeline:heartbeat --pulse 2

How to configure the runners

Runners are doing the job
They execute the primary logic of each step if the step is ready to run
You need to have at least one runner-up and running
It would help if you had multiple runners to use the multiprocessing capabilities of Pacemaker

How to execute a runner

A Pacemaker runner is a message queue consumer. Therefore you need to run the default Magento queue:consumers:start command to start a Pacemaker pipeline runner.

bin/magento queue:consumers:start pipeline.runner.amqp

There are two ways to control the Magento Consumer / PipelineRunner

SuperVisor
Magento Cron-Jobs for consumers

Regardless of which option is chosen, it has to be ensured that the other one is deactivated.

Using supervisor for Pacemaker runners

With supervisor it is possible to run multiple runners and ensure that all runners are up and running all the time.

See following supervisor configuration example:

[program:pipelineRunner]
stderr_logfile = <MAGENTO_ROOT>/var/pipelineRunner.log
autorestart = 1
autostart = 1
startsecs = 0
startretries = 0
directory = <MAGENTO_ROOT>
numprocs = 4
process_name = %(program_name)s_%(process_num)02d
user = www-data
command = /usr/bin/env php <MAGENTO_ROOT>/bin/magento queue:consumers:start --max-messages 1 pipeline.runner.amqp

Option Description

Option	Description
`--max-messages 1`	Limit runner messages (`--max-messages 1`). We recommend using the `--max-messages` option with value `1` to avoid issues within stateful PHP code It depends on how you implement your job executors, but there is still vendor code, which could have a state which would raise issues that are easy to fix but hard to detect.
`startsecs`	startsecs config option. This option is required since we have executors running faster than one second. In this case, the supervisor would stop restarting the runners. See the supervisor configuration documentation for more details
`numprocs`	numprocs config option. This option is defines the concurrent processes of the given command. We recommend 4 - 5 runners based on the cpu core count.

--max-messages 1

Limit runner messages (--max-messages 1).

We recommend using the --max-messages option with value 1 to avoid issues within stateful PHP code

It depends on how you implement your job executors, but there is still vendor code, which could have a state which would raise issues that are easy to fix but hard to detect.

startsecs

startsecs config option.

This option is required since we have executors running faster than one second. In this case, the supervisor would stop restarting the runners.

See the supervisor configuration documentation for more details

numprocs

numprocs config option.

This option is defines the concurrent processes of the given command.

We recommend 4 - 5 runners based on the cpu core count.

See following env.php configuration example to disable cron consumer runners:

...
'queue' => [
        ...
        'consumers_wait_for_messages' => 1
    ],
...
    'cron_consumers_runner' => [
        'cron_run' => 0,
        ...
    ],
...

Using cron management for Pacemaker runners

With cron consumer runners it is possible to run multiple runners and ensure that all runners are up and running at application side. This is recommended for running Pacemaker at Adobe Commerce Cloud Instances.

See following env.php configuration example to run Pacemaker with cron consumer runners:

...
'queue' => [
        ...
        'consumers_wait_for_messages' => 1
    ],
    ...
    'cron_consumers_runner' => [
        'cron_run' => 1,
        'max_messages' => 100,
        'max_idle_time' => 3600,
        'consumers' => [
            'pipeline.runner.amqp'
        ],
        'multiple_processes' => [
            'pipeline.runner.amqp' => 4
        ]
    ],
...

Option Description

Option	Description
`consumers_wait_for_messages`	Specify whether consumers should continue polling for messages if the number of processed messages is less than the `max_messages` value (default is `1`). The `consumers_wait_for_messages` option is a global option and cannot be configured separately for each consumer. Adobe Commerce Dev-Docs
`cron_run`	A boolean value that enables or disables the `consumers_runner` cron job (default is `1`).
`max_messages`	The maximum number of messages each consumer must process before terminating.
`max_idle_time`	Get maximal time (in seconds) for waiting new messages from queue before terminating consumer.
`consumers`	An array of strings specifying which consumers to run. An empty array runs ALL consumers.
`multiple_processes`	An array of key-value pairs specifying which consumer to run in how many processes. Supported in Adobe Commerce 2.4.4 or greater.

consumers_wait_for_messages

Specify whether consumers should continue polling for messages if the number of processed messages is less than the max_messages value (default is 1).

The consumers_wait_for_messages option is a global option and cannot be configured separately for each consumer.

Adobe Commerce Dev-Docs

cron_run

A boolean value that enables or disables the consumers_runner cron job (default is 1).

max_messages

The maximum number of messages each consumer must process before terminating.

max_idle_time

Get maximal time (in seconds) for waiting new messages from queue before terminating consumer.

consumers

An array of strings specifying which consumers to run. An empty array runs ALL consumers.

multiple_processes

An array of key-value pairs specifying which consumer to run in how many processes. Supported in Adobe Commerce 2.4.4 or greater.

To learn more about the cron consumer configuration, please read the Adobe Commerce Dev-Docs

It is recommended to specify and set the runtime of the longest pipeline steps for consumer_timeout configuration.

The timeout value is configurable in rabbitmq.conf (in milliseconds) e.g.:

# 6 hours in milliseconds
consumer_timeout = 21600000

If a consumer does not ack its delivery for more than the consumer_timeout value (30 minutes by default), its channel will be closed with a PRECONDITION_FAILED channel exception.

The error will be logged by the node that the consumer was connected to. All outstanding deliveries on that channel, from all consumers, will be requeued.

See Per-node Configuration for more information in RabbitMQ Docs.

It is not recommended to run multiple consumers on a MySQL-operated queue.

See Change message queue from MySQL to AMQP for more information.