Caching & cache warming

The Pacemaker Professional Edition cache warming functionality offers advanced caching features for this purpose and the possibility to warm up caches to increase speed.

  • Caching in an import scenario cannot significantly improve the speed

  • At least in a cloud environment, savings are possible by keeping database queries to a minimum, as these queries get processed more slowly there than on a dedicated server

The cache is divided into two types, reducing the amount of data transferred:

  • Static cache

  • Configurable cache

Static cache

  • The static cache can not be disabled, as it gets used for caching global data and the artifacts created during the import process

  • Avoid unnecessary database queries when variants get created after simples have got created

Configurable cache

  • The cache is disabled by default to avoid uncontrolled memory usage

  • The configurable cache can get enabled on the console by adding the parameter --cache-enabled to one of the import commands

Console

Example

Enable the configurable cache with a CLI command:
vendor/bin/import-pro import:products --cache-enabled=true

Enable/Disable cache

In certain conditions, it is useful to enable caching, especially in scenarios where Pacemaker needs to operate in a distributed environment, such as on AWS.

To enabled/disabled the configurable cache or change the TTL without adding the console parameter every time it will be invoked.

Example

Add the snippet <magento-install-dir>/app/etc/configuration/cache.json depending the requirements:
{
  "caches": [
    {
      "type": "cache.static" (1)
    },
    {
      "type": "cache.configurable", (2)
      "enabled" : true, (3)
      "time": 1440 (4)
    }
  ]
}
1 Static cache section
2 Configurable cache section
3 Set "enabled" : "true|false"
4 The parameter time adjusts the TTL of the cache entries and should be changed only when there are good reasons for it

It makes no sense to disable the cache when the cache warming functionality has got enabled because the loading depends on the size of the database; the data for the cache warming functionality can take some time. Still, the data itself will not be cached and can not be used to reduce database access in the end.

Cache warming

By enabling the cache warming functionality, Pacemaker loads as much data as possible into the cache using optimized queries. Thatreduces the database access during the import process.

Cache warming, therefore, has the most significant impact on performance, as all products and their attributes and many relations are preloaded, and database queries are no longer necessary.

The great advantage of cache warming faces an enormous memory requirement because this data must be kept in memory until the import process gets completed

Examples

Default cache-warmer configuration:
{
  "operations": {
    "general": {
      "catalog_product": {
        "cache-warmer": {
          "plugins": {
            "cache-warmer": {
              "id": "import_caching.plugin.cache.warmer",
              "params": {
                "cache-warmers": [
                  "import_caching.repository.cache.warmer.eav.attribute.option.value",
                  "import_caching.repository.cache.warmer.product",
                  "import_caching.repository.cache.warmer.product.varchar",
                  "import_caching.repository.cache.warmer.product.int",
                  "import_caching.repository.cache.warmer.product.text",
                  "import_caching.repository.cache.warmer.product.decimal",
                  "import_caching.repository.cache.warmer.product.datetime"
                ]
              }
            }
          }
        }
      }
    }
  }
}

  • The Pacemaker provides cache-warmers for the product entity as well as the EAV attribute option values.

To enable the cache warming functionality, assuming you are using Magento Community, simply add the appropriate operation general/catalog_product/cache-warmer to a snippet like <magento-install-dir>/app/etc/configuration/shortcuts.json with the shortcuts which could then look like

To enable cache-warming functionality, if you are using Magento Community, add the appropriate operation general/catalog_product/cache-warmer to a snippet like, e.g., at <magento-install-dir>/app/etc/configuration/shortcuts.json with the shortcuts, see the following snippet

{
  "shortcuts": {
    "ce": {
      "catalog_product": {
        "add-update": [
          "general/general/global-data",
          "general/general/move-files",
          "general/catalog_product/cache-warmer",
          "general/catalog_product/collect-data",
          "general/eav_attribute/convert",
          "general/eav_attribute/add-update.options",
          "general/eav_attribute/add-update.option-values",
          "general/eav_attribute/add-update.swatch-values",
          "general/catalog_category/convert",
          "ce/catalog_category/sort",
          "ce/catalog_category/add-update",
          "ce/catalog_category/add-update.path",
          "ce/catalog_category/add-update.url-rewrite",
          "ce/catalog_category/children-count",
          "ce/catalog_product/validate",
          "ce/catalog_product/add-update",
          "ce/catalog_product/add-update.variants",
          "ce/catalog_product/add-update.bundles",
          "ce/catalog_product/add-update.links",
          "ce/catalog_product/add-update.grouped",
          "ce/catalog_product/add-update.media",
          "general/catalog_product/add-update.msi",
          "general/catalog_product/add-update.url-rewrites"
        ]
      }
    }
  }
}

The cache warming functionality is only helpful in the add-update operation, as the delete and replace operations do not use pre-cached data because of their implementation nature.

Finders

As part of Pacemaker Professional Edition, the techdivision/caching library contains replacements for the generic finder implementations which use the configurable cache type.

Therefore, the default finders are overridden by DI and use the configurable cache to cache the result of database queries.

  • It ensures that the cache is invalidated when existing data has got replaced

{
  "finder-mappings": {
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DATETIMES":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DECIMALS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_INTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_TEXTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHARS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT":
    "import_caching.repository.finder.factory.unique.entity.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DATETIMES_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DECIMALS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_INTS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_TEXTS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHARS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHAR_BY_ATTRIBUTE_CODE_AND_ENTITY_TYPE_ID_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHAR_BY_ATTRIBUTE_CODE_AND_ENTITY_TYPE_ID_AND_STORE_ID_AND_VALUE" :
    "import_caching.repository.finder.factory.unique.cached"
  }
}

Each of these finders replaces a version that does not use the cache.

The finders can therefore be used for a fine-grained cache configuration, allowing the memory utilization of Pacemaker to be adjusted during the import process.

Fine-grained cache configuration

As described, cache warming can have a significant impact on memory consumption.

In combination with the configuration of the finders to get used, it is possible to adjust the memory consumption to your own needs.

The snippet <magento-install-dir>/app/etc/configuration/operations.json overrides the default cache warmers
Therefore the snippet <magento-install-dir>/app/etc/configuration/operations.json overrides the
default cache warmers and will remove the cache warmer "import_caching.repository.cache.warmer.product" for the products.

{
  "operations": {
    "general": {
      "catalog_product": {
        "cache-warmer": {
          "plugins": {
            "cache-warmer": {
              "id": "import_caching.plugin.cache.warmer",
              "params": {
                "cache-warmers": [
                  "import_caching.repository.cache.warmer.eav.attribute.option.value",
                  "import_caching.repository.cache.warmer.product.varchar",
                  "import_caching.repository.cache.warmer.product.int",
                  "import_caching.repository.cache.warmer.product.text",
                  "import_caching.repository.cache.warmer.product.decimal",
                  "import_caching.repository.cache.warmer.product.datetime"
                ]
              }
            }
          }
        }
      }
    }
  }
}

Example

Additionally, a snippet like <magento-install-dir>/app/etc/configuration/finder-mappings.json, that overwrites the default finder configuration and replaces the cache finder implementation import_caching.repository.finder.factory.unique.entity.cached for the (SQL) statement TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT with a non-cached version import.repository.finder.factory.unique.

See the following snippet example how the snippet prevents Pacemaker from caching product entities:
{
  "finder-mappings": {
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DATETIMES":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DECIMALS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_INTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_TEXTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHARS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCTS":
    "import.repository.finder.factory.yielded",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT":
    "import.repository.finder.factory.unique",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DATETIMES_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_DECIMALS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_INTS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_TEXTS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHARS_BY_PK_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHAR_BY_ATTRIBUTE_CODE_AND_ENTITY_TYPE_ID_AND_STORE_ID":
    "import_caching.repository.finder.factory.yielded.cached",
    "TechDivision\\Import\\Product\\Utils\\SqlStatementKeys::PRODUCT_VARCHAR_BY_ATTRIBUTE_CODE_AND_ENTITY_TYPE_ID_AND_STORE_ID_AND_VALUE" :
    "import_caching.repository.finder.factory.unique.cached"
  }
}

  • If only the cache warmer gets removed, the products will get loaded into the cache

  • Instead, all at once, what gives a significant performance boost, one after another

  • It will result in the same level of memory consumption but with lower performance

WIR HÄTTEN IHNEN DIESE MELDUNG AUCH GERNE ERSPART ...

... aber wir müssen Sie informieren, dass wir auf unserer Website Cookies verwenden:
  1. Diese Website verwendet keine technisch notwendigen Cookies.
  2. "Böse" Tracking-Cookies von Google Analytics (natürlich mit IP-Anonymisierung) und Leadfeeder, damit wir unsere Inhalte besser auf Ihre Bedürfnisse anpassen können.
Solange Sie nicht explizit akzeptieren, tracken wir Sie auch nicht.
Versprochen!

Detaillierte Infos in unserer Datenschutzerklärung

P.S.: Während der Entwicklung dieser Website wurden intensiv Chocolate Cookies eingesetzt.
Die sind aber alle schon weg - wir bräuchten bitte neue.
Akzeptieren Verweigern