PIM integrations and architecture: your questions answered

What this episode covers

The eighth episode is the longest conversation in the series and the only one with a technical guest. Karolina asks, and Marcin Żebrowski, senior Symfony developer at Macopedia, answers the questions clients raise most: how an integration works, what middleware is, where to keep images, who provides the server, and what can be added to the system.

Key takeaways

• Three ways to integrate: direct when the code is open, middleware, or an off-the-shelf plugin. One rule: ready-made before custom.
• A direct integration ties you to the platform. An upgrade means retesting everything, and moving to another system becomes hard.
• Sync is usually once a day; it can be every 15 minutes, but frequent transfers load the system. A hybrid with a manual trigger works best.
• Do not host images straight from a PIM system. PIM systems are not built for heavy traffic; a CDN serves the images while PIM keeps the original.
• When customising, prefer adding a feature next to the platform, in middleware, over hooking into its code, which can break on upgrade.

Where the time saving comes from

The warm-up question: how does a PIM system actually speed up work with products? The answer is simple and not technological at all. The data is in one place.

In many companies information sits in different spreadsheets in different departments, and departments do not know that the information they need already exists in a spreadsheet elsewhere. Gathering it in one system gives easy access, and only on that can you build integrations with ERP, e-commerce, marketplaces and a DAM system.

What middleware is

Middleware is, as the guest puts it, a command centre: an application in the middle, between the PIM system, e-commerce and everything else. The flow looks like this: a user adds data in the ERP, middleware pulls it, transforms it and writes it into the PIM system. In the other direction it takes data from PIM, transforms it and sends it to e-commerce.

Three ways to integrate

Direct. Possible when the system has open code or allows custom elements. The advantage is less development work. The drawback, and a serious one, is being tied to the platform: upgrading to a higher version means retesting everything, some things will stop working, and moving to a different system becomes very hard.

Middleware. Necessary with closed-source SaaS systems hosted by the vendor. Worth knowing that it is not always built from scratch. Every project is checked for a low code platform, for example Mage AI, where you get ready-made flows and error logging and only add specific fragments, such as mapping one attribute to another.

A plugin. A small ready application you install and configure. Upside: you do not build the integration from the ground up. Downside: you cannot modify the plugin to fit you, so you adapt to it. Often, though, the changes on the organisation’s side are small enough to be worth it and save a lot of time.

The recommendation is: find a ready-made solution and deploy it. Write your own code only when nothing ready exists. A mix is fine, with a bias towards ready-made.

How often data synchronises

Usually automatically, once a day, for example at midnight when nobody is working. You can set every 15 minutes, but frequent transfers load the system and may cause slowdowns.

The hybrid works best: the nightly automation plus the option to trigger the integration manually, for when somebody needs a change to appear in the shop immediately. From the team’s experience: that option ended up in practically every project.

What an integration costs and how to shorten it

Cost depends on complexity. The example in the episode is telling: an integration with a data bank that required moving around 300 attributes. At half an hour per attribute that is roughly 19 working days. Add the number of integrations and whether the system on the other side already works. Once, integrating with a system being rolled out in parallel at the client, part of the functionality did not work, there was no documentation, and the timeline stretched.

Ways to shorten it: reusable middleware components carried between projects, official packages for popular systems, plugins where they fit, and prioritising during the pre-implementation analysis, meaning only the key integrations at the start.

How to prepare

Three steps, in this order.

An attribute audit. Gather everything you have in one place: in spreadsheets, in the ERP, in a DAM, on drives. First you need to know what exists.

Deciding which data goes where. There is no point sending all 300 attributes to a system that will use five. You would have to manage them, and the integration costs money.

Mapping. Preparing data for loading and for onward transfer. If the PIM system holds numeric data, metric data and images, and the target system takes only text and images, it has to be transformed. That is the foundation of what middleware does.

Images: PIM, DAM or CDN

The question is whether a PIM system can act as a DAM. It depends on the need. For storage, light edits and distribution, PIM systems are enough. If you need image scaling, thumbnail generation and broader processing, choose a dedicated DAM.

The most important part, though, is how images are served. PIM systems can publish them straight to a site, but that is not a good solution, because they are not built for heavy traffic. The recommendation: PIM holds the master, high-quality image as the source of truth, and a CDN serves it, resizing on the fly, built for intensive traffic.

Data formats and migration

The key is translating everything into a common language. Spreadsheets load through configurators mapping columns to attributes. Data from other systems is pulled via API and mapped by transformers on our side. In the other direction: an export to a spreadsheet in the format the target system expects, or a transformation in middleware and a transfer over API.

For the first load the team uses a method the guest calls brute force outright: the client fills in a prepared template with attributes from their own spreadsheets, an import is attempted, errors come back, and scripts are written from those errors to fix whole files. After a few iterations the import succeeds. Why this rather than integrating with every client system? Because it is a part of the work the client can take on, and it comes out cheaper and faster than building integrations purely for the migration.

Hosting

It depends on the system. Free community editions are open source and the vendor does not provide a server: the server comes from the implementation partner or the client, and installation is part of the project. In the SaaS model the vendor installs and maintains it, with no access to code or server, and you get a ready platform.

In practice that means Akeneo Enterprise and Ergonode go the SaaS route, while Akeneo Community and Pimcore Community run on your own server.

Customisation

The order is always the same: check the system’s app store first, because ready-made is cheaper than new.

If nothing ready exists, with open code the only limit is budget, but bear in mind a custom plugin may stop working after a system upgrade. That is why the better route is usually adding the feature next to the platform, in middleware: pull the data, modify it, write it back. Of the three systems discussed, Pimcore is the most geared towards such modifications.

One more observation worth remembering: after digging into the real need it often turns out that the system’s existing features are enough, or that moving to a higher licence tier where the feature already exists is cheaper than building it.

Have a technical question about your implementation? Get in touch. More on the PIM systems page and on system integrations.