Service Description - Pacemaker Import Pipelines 5.x

Technical Component Informationen

Company

TechDivision GmbH

Name

Pacemaker Import Pipelines

Version

5.x

Web

https://pacemaker.met.tdintern.de/components-pacemaker-import-pipelines/v5.x/index.html

Open Source

License

Proprietary

Pricing

n/a

Adobe Commerce Version

>= 2.4.4

PHP Version

>= 7.4

General

Although Magento already has an import functionality of products in the backend, it is often necessary to perform imports in a more performant and automated way.

Pacemaker Import Pipelines is part of Pacemaker Enterprise Edition and includes standard import pipelines for data and extensions in terms of configuration and processing logic.

The Pacemaker Import Pipelines uses the Pacemaker Process Pipelines Framework (Process Pipeline component) and the Import Framework (Import Pro component) and extends it with the following core functions:

Supply of standardized import pipelines
- Catalog import
  - Attribute sets
  - Attributes
  - Categories
  - Products
- Media Import
- Price Import
- Inventory Import
- Customers Import
- Customer Address Import
- Gift Card Import
Improve Magento index behavior with Pacemaker indexer
Provide advanced pipeline and step conditions for use in custom pipelines
Api-driven Imports

Feature Overview

Function	Description
Import catalog data	The component provides multiple standard pipelines available for: Importing catalog data Attributes Attribute sets Categories Products These standard pipelines can be modified as desired or used as the basis for custom import pipelines.
Importing media (images)	The component provides a standard pipeline for importing media, e.g., product images. This import has additional features such as image cache invalidation and image-cache generation.
Import of price information	The component provides standard pipelines to import price data. Prices/Special prices Tier prices
Import of inventory information	The component provides standard pipelines to import inventory data. Quantity-Inventory Import Stock and Source Inventory Import (MSI)
Import of customer data	The component provides a standard pipeline to import customer data.
Import of customer address data	The component provides a standard pipeline for importing address data.
Import of gift cards Information	The component provides a standard pipeline to import gift cards
Indexer logic control	The component extends the control of the standard Magento Indexer logic with the Pacemaker Indexer Pipeline.
Improve indexer performance	Controlling index operations with Pacemaker Indexer improves the runtimes of index operations after import.
Control of dependencies between import and indexing	Controlling imports and index operations through pipelines allows dependencies to be created and conflicts to be eliminated.
Performant Api Imports	Optional Bulk-Api-Path to import large data with Pacemaker via api call.
Api-driven data import	Additional REST-CSV-Api to import standard data

Function

Description

Import catalog data

The component provides multiple standard pipelines available for:

Importing catalog data
Attributes
Attribute sets
Categories
Products

These standard pipelines can be modified as desired or used as the basis for custom import pipelines.

Importing media (images)

The component provides a standard pipeline for importing media, e.g., product images.

This import has additional features such as image cache invalidation and image-cache generation.

Import of price information

The component provides standard pipelines to import price data.

Prices/Special prices
Tier prices

Import of inventory information

The component provides standard pipelines to import inventory data.

Quantity-Inventory Import
Stock and Source Inventory Import (MSI)

Import of customer data

The component provides a standard pipeline to import customer data.

Import of customer address data

The component provides a standard pipeline for importing address data.

Import of gift cards Information

The component provides a standard pipeline to import gift cards

Indexer logic control

The component extends the control of the standard Magento Indexer logic with the Pacemaker Indexer Pipeline.

Improve indexer performance

Controlling index operations with Pacemaker Indexer improves the runtimes of index operations after import.

Control of dependencies between import and indexing

Controlling imports and index operations through pipelines allows dependencies to be created and conflicts to be eliminated.

Performant Api Imports

Optional Bulk-Api-Path to import large data with Pacemaker via api call.

Api-driven data import

Additional REST-CSV-Api to import standard data

Pacemaker Indexer

Pacemaker Indexer improves the runtimes and traceability of indexing runs.

Magento standard functions are still used for the indexing process itself; only the control has been adapted.

Instead of execution with cron logic, a process pipeline is created and processed. That has several advantages over cron processing.

Temporal controllability with configurable schedules Ability to set dependencies, which gets already used for import runs Performance optimization of indexer runs after imports due to identification of index delta and status Possibility to block indexing (not recommended, developer tool only)

Media-Cache-Management

In the standalone media import pipeline and as an extension of the product import pipeline, two steps get provided to handle image cache cleanup.

The first step is responsible for the invalidation of the image cache, and the second step is responsible for the regeneration of the cache.

Both steps can be controlled individually by configuration.

Validation

In addition to data import via the file system, the components offer two Api endpoints.

Firstly, a CSV file api to store the import files independently of the file system access.

Secondly, an alternative bulk api to import or update large amounts of data entities more efficiently than Adobe Commerce.

Import Api

Various validation functions are performed during import processes to recognise and output errors or warnings.

In addition, it is possible to run a dedicated validation pipeline to check data without changing the data status of the Adobe Commerce instance.

Expandability / Extension Points

As listed, the component provides many concepts and code-based extension points for extending and customizing imports and pipelines.

Therefore, it is easy to create custom import pipeline definitions.

See https://pacemaker.met.tdintern.de/components-pacemaker-import-pipelines/v5.x/example-import-pipeline/index.html

What is included in the license price?

Major/Minor Updates
Security Updates
Bugfixes

What the component cannot do

The component does not provide any export functions. Only standardized executable import pipelines get provided
The component does not provide automated (plug&play) connections to third-party ERP systems (e.g., SAP) , only the processing of provided CSV gets provided.

Requirement for use

To use the Pacemaker Components Import Pipelines in Adobe Commerce, the following points must get observed:

Contract
Technical conditions

Glossary

Function Description

Function	Description
Pipeline	Pipeline is the chaining of commands.
Step	A step is understood to be a chain link within a pipeline.
Index	An index represents an index structure in a database, separate from the data structure, that speeds up searching and sorting by specific fields. Indexes are used to provide quick access to data in a large collection of data
Indexer	An indexer performs the indexing process to create the index structure Magento requires. See `https://developer.adobe.com/commerce/php/development/components/indexing/`
Media Cache	Magento uses an image cache to display product images. Product images are created for the corresponding screen sizes during caching, depending on the settings. Therefore, the size adjustment/optimization does not have to be done for each call.
MSI	With the Multi Stock Inventory (MSI), Magento offers extended functionalities for stock management in Magento 2 since Magento 2.3.
Cron	CronJobs are repeating tasks that can be executed automatically on Unix-based operating systems like Linux or iOS/Mac OS X and on server environments. The acronym Cron stands for `command run on notice`.
Bulk endpoints	Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue.. See `https://developer.adobe.com/commerce/webapi/rest/use-rest/bulk-endpoints/`

Pipeline

Pipeline is the chaining of commands.

Step

A step is understood to be a chain link within a pipeline.

Index

An index represents an index structure in a database, separate from the data structure, that speeds up searching and sorting by specific fields.

Indexes are used to provide quick access to data in a large collection of data

Indexer

An indexer performs the indexing process to create the index structure Magento requires.

See https://developer.adobe.com/commerce/php/development/components/indexing/

Media Cache

Magento uses an image cache to display product images.

Product images are created for the corresponding screen sizes during caching, depending on the settings.

Therefore, the size adjustment/optimization does not have to be done for each call.

MSI

With the Multi Stock Inventory (MSI), Magento offers extended functionalities for stock management in Magento 2 since Magento 2.3.

Cron

CronJobs are repeating tasks that can be executed automatically on Unix-based operating systems like Linux or iOS/Mac OS X and on server environments.

The acronym Cron stands for command run on notice.

Bulk endpoints

Bulk API endpoints differ from other REST endpoints in that they combine multiple calls of the same type into an array and execute them as a single request. The endpoint handler splits the array into individual entities and writes them as separate messages to the message queue..

See https://developer.adobe.com/commerce/webapi/rest/use-rest/bulk-endpoints/