Indexer pipeline

The Pacemaker indexer provides a new pipeline with the following steps:

Using application lock mode for reindex processes

According to the Dev-Docs, the status values in the indexer_state or mview_state database tables may not be the same as what is observed, because they sometimes do not get updated when an indexer fails.

You can enable use_application_lock mode for reindexing through the use of environment variables or by configuring:

Extract from a env.php (app/etc/env.php)

return [
    'indexer' => [
        'use_application_lock' => true
    ],
    ...
];

In case of a failure during the reindexing of a certain indexer, having use_application_lock mode enabled will return a more accurate status of the indexer.

Configure the expiration interval for the pipeline

In rare cases, indexer pipelines maid freeze and block other pipelines from executing.

To avoid indexer pipelines get stuck and blocking other pipelines, a pipeline gets equipped with the expire-in XML attribute::

Extract from a pipeline.xml (vendor/techdivision/pacemaker-indexer/etc/pipeline.xml)

<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:TechDivision_ProcessPipelines:etc/pipeline.xsd">
    <pipeline name="pacemaker_indexer" description="Pacemaker Indexer Pipeline"
              expire-in="6 hours"> (1)
        ...
    </pipeline>
</config>

1	`expire-in` XML attribute: Sets the expiration interval for the pipeline (in the example, it is set to `6 hours`) After this interval, the status of the pipeline and all its steps will get changed to `canceled` Possible values of attribute is `[0-9]* (minutes\|hours\|days\|weeks\|months\|years)`

Configure how consumer process messages from the message queue

According to the Dev-Docs, in some cases, to prevent issues caused by long delays in message queue processing can be used following env.php configuration::

...
'queue' => [
    'consumers_wait_for_messages' => 0
    ...
],
...

This configuration can lead to improvements when the pipeline freezes and blocks other pipelines

As a last resort, is it possible to cancel given pipelines with CLI and abort all running processes.

Force Reindex

The first step of the indexer pipeline is the step pacemaker_force_reindex.

The associated executor IndexerForceReindexExecutor uses the indexer-PurposeChain to check whether a full reindex should be triggered for an index
Only held or set indices, which the Index Setter has prepared, are checked

Detail view Force-Reindex-Step (Magento-Adminhtml)

Reindex all invalid indices (delta behaviour)

The second step of the indexer pipeline is the step pacemaker_reindex_all_invalid.

The associated executor IndexerReindexAllInvalidExecutor only executes the Magento standard logic and updates the invalid indices.

The advantage of outsourcing the function to an executor is the management of dependencies

Detail view Reindex-All-Invalid-Step (Magento-Adminhtml)

Import Reindex / Update mViews (delta behaviour)

The third step of the indexer pipeline is the step pacemaker_import_reindex.

The associated executor IndexerControlExecutor only executes the Magento standard logic and updates the view tables (mViews)

The advantage of outsourcing the function to an executor is the management of dependencies
Also the management and performance improvement for the indexing flows with the Indexer - PurposeChain

Detail view Import-Reindex-Step (Magento-Adminhtml)

Fix Indexer status

The fourth step of the indexer pipeline is the step pacemaker_indexer_fixer.

The associated executor IndexerFixer fix incorrect indexer status caused by an error during indexing. This step is optional and can be enabled or disabled in Admin.

In this step, the index status will be changed to valid or invalid (as configured in PACEMAKER Pipeline Settings Indexer Settings Fixed Indexer Status) if all conditions are met:

This step is enabled in PACEMAKER Pipeline Settings Indexer Settings Enable Indexer Fixer.
At least one of the first three steps (pacemaker_force_reindex, pacemaker_reindex_all_invalid or pacemaker_import_reindex) has one of the following statuses error, abandoned or canceled.
This index has the status working.

Detail view Fix indexer status (Magento-Adminhtml)

Clear index changelogs (delta behaviour)

The fifth step of the indexer pipeline is the step pacemaker_clear_indexer_changelog.

The associated executor IndexerClearChangelog only executes the Magento standard logic and clean up the changelog information of indices.

The advantage of outsourcing the function to an executor is the management of dependencies

Detail view Clear-CL-Step (Magento-Adminhtml)

Cleanup

The last step of the indexer pipeline is the step pacemaker_indexer_cleanup.

This step is primarily used to clear the index pipelines, because depending on configuration an indexer pipeline is created every minute, which may lead to enormous amounts of data and make the pipeline view in the backend confusing.

You can configure how long indexer pipelines are allowed to exist after they have been processed

Detail view indexer-Cleanup-Step (Magento admin HTML)

Delta Indexer Command

The delta indexer can be executed using the pacemaker:index:delta command. First, it retrieves all validated groups. Then, it checks if a full reindex is necessary. The indexer collects the threshold values from the config, gathers all incorrect indices, and checks if the number of incorrect indices exceeds the threshold value from the config. If this is the case, the delta indexer command executes a full reindex. otherwise, it executes a delta index. The default threshold is 1000 for all indices except the customer grid and the design config grid indices.

Indexer pipeline

Using application lock mode for reindex processes

Configure the expiration interval for the pipeline

Configure how consumer process messages from the message queue

Force Reindex

Reindex all invalid indices (delta behaviour)

Import Reindex / Update mViews (delta behaviour)

Fix Indexer status

Clear index changelogs (delta behaviour)

Cleanup

Delta Indexer Command

WIR HÄTTEN IHNEN DIESE MELDUNG AUCH GERNE ERSPART ...