Insenio

In 2018 I joined Insenio GmbH, a small Berlin-based startup running a WooCommerce online-shop. The company was founded in 2015 and had already established a loyal customer base by the time I joined as IT Lead.

In this role I am mainly responsible for:

• ensure smooth operation
• improve website performance
• make technology and infrastructure decisions
• scale IT infrastructure, and migration to new platforms
• IT security and GDPR compliance
• development of new features
• offer technical solutions for marketing and support
• implement new payment methods
• integrate with marketplace APIs
• launch new (brand) shops
• add new markets and languages to the shops
• coordinate and mentor external freelancers and make code reviews

Baseline or Status Quo

The existing online shop was realised with WordPress & WooCommerce. Code was managed in a Git repository and deployed automatically. An external web designer did maintenance work. A tech-savvy person from marketing had added many WordPress and WooCommerce plugins to implement certain things without programming. Unfortunately this lead to extra complexity as non-standard states had been added and decreased page load times due to 60+ plugins being used. A monolithic WordPress & WooCommerce setup on a single web server with many single points of failure such as Elasticsearch, PHP and MySQL all installed alongside each other.

WordPress & WooCommerce task and challenges

Here is list of exemplary tasks and challenges, and the solutions I provided to Insenio GmbH to solve these issues. All of these tasks are basically aimed at ecommerce operations run smoothly with a blogging platform.

Slow Order and Customer Searches

Problem: The standard WooCommerce order search in the backend was used by support staff to find orders. When customers called support, they were not able to find orders, emails quickly enough. Customers often had to wait on the phone while support staff waited for search results.

Cause: The reason for slow searches lies in WordPress’ inefficient database design. WordPress stores most data into two tables posts and postmeta which can get very huge in size quickly. Also, WooCommerce did a full text search across too many columns of unindexed tables.

Solution: After identifying all fields customer support searches orders and customers by, I developed a custom WordPress order search plugin. The plugin introduced an optimised and indexed search table which is fast and easy to query against. The table was automatically filled with data via MySQL INSERT & UPDATE triggers on the posts and postmeta tables. Only one indexed, concatenated string column was used to find matches. The search form fires an AJAX request, avoiding unnecessary reloading of the search page. This reduced search time from 10-30 seconds to about 20-50 milliseconds.

Slow Loading Trusted Shops Reviews

Problem: product reviews made by customers on Trusted Shops were loading slow, which resulted in the layout container visibly “jumping”. Also, only new product reviews from the last year are displayed.

Solution: instead of Trusted Shops standard way of displaying reviews using JSONP via additional HTTP request to a Trusted Shops server, I built a WordPress plugin with CLI-command which retrieves new product and shop reviews from their API using a Cronjob. New reviews are then stored to the database and can thus be rendered in the shop templates directly and indefinitely.

Server Migration and general Codebase improvements

The large number (60+) WordPress and WooCommerce plugins in use needed to be reduced in order to improve overall site performance. I refactored the deploy process using Github Actions. Unused, old and small plugins were removed and where possible their functionality was implemented in the theme.

To improve overall site performance I did a server migration from a single server to a server cluster with dedicated Load Balancer, App and MySQL servers. This migration included allowed to apply many best practices and upgrades such as:

• Upgrade WordPress and WooCommerce to recent versions
• Upgrade PHP7 from v7.0 to v8.0
• Upgrade MySQL from v5.7 to a dedicated server with v8.0
• Normalize database encodings and add multibyte support
• Replace Apache with Nginx Server and Reverse Proxy
• Make use of HTTP 2.0 (before HTTP 1.0)
• Introduce Composer package manager
• Add automatic Backups for User Uploads and Databases
• Introduce a CDN for assets
• Add Nginx Amplify and New Relic APM

This improved overall performance and load times.

Launch Orgawell Online Shop

In 2019, Insenio acquired the Switzerland-based brand Orgawell. In order for all team members (support, marketing and external web designers) to be able to to work with the shop, we decided to use WordPress & WooCommerce for this shop. The relaunch was done for Germany, Switzerland and Austria (DACH) regions allowing for individual domains, content, payment methods and pricing matrixes.

Many learnings from Insenio, best practices as well as pre-made customisations and plugins could be reused and perfected in this project, which sped up the launch. The design was based on the pre-made Ecommerce Theme Flatsome.

While running on a single server and having a way smaller database, I implemented many performance tweaks. The site did not require a full page cache, but made use of some performance Plugins such as Perfmatters and Redis Object Cache. Dependencies were managed with composer, deploys made with Github Actions and it always used the latest versions of WordPress and WooCommerce.

Warehouse processes were adjusted for a family-run Warehouse close to the Switzerland border.

Launch Insenio France & Belgium

In 2020, Insenio expanded into the French speaking market. Therefor French support staff and content editors were hired. I was in charge of creating a WooCommerce satellite store.

The shops were realised in a standalone WordPress & WooCommerce installation but were connected to the main (German) sites. While the site differs in language, pricing matrix, payment and shipping methods, deliveries were made from the same warehouse.

This meant that this satellite store had to consume and update product stock in the remote “master” shop and that orders were sent to the same warehouse thus needed a different number range to prevent collisions of incremental ranges. Stock updates were done via WooCommerce API and read-only database user, with a short-lived microcache to reduce the number of requests.

Initial content translations were done via the Deepl API, Translators only had to correct and approve content.

Migration to Shopware 6

Due to constant growth and expansion, the risk of overselling products increased, as some sales channels and marketplaces (Amazon, Shop Apotheke, French website, etc.) needed to sync product stock. Especially marketplaces, and other marketing tools which consume product feeds were increasingly difficult to handle across several installations.

In 2021 we saw further decreases in site performance due to WordPress inefficient database design. While page rendering remained extremely fast (thanks to caching) – all database writes got intolerantly slow. The only solution to this bottleneck was a full relaunch with real shopping platform that stores data semantically. After evaluating several platforms a decision was made to use the brand new Shopware 6, which had recently made a shift to modern web technologies. Also, the availability of pre-made plugins for 3rd party services already in use by Insenio and its compliance with GDPR were factors.

Goals

• using an actual shopping platform, be future proof
• improve performance
• eliminate single points of failure
• improve data consistency and prevent oversells
• faster time to market for expansion to more countries, languages and marketplaces
• improved infrastructure scalability and redundancy
• making use of a fully responsive theming engine

Challenges

Importing millions of records (orders, customers, addresses, products, etc) is tricky by itself, but even more so when Shopware comes with many different ways to import data (Import Wizard, CSV, API, Data Fixtures, Migrations, Programmatically) and when one method has shortcomings or can only be done if the old platform knows about the new platforms UIDs.

After initial data imports in testing environments, we noticed that – similar to WooCommerce – order and customer searches were unacceptably slow. While adding database indexes helped only a little bit, we were forced to program a custom Elasticsearch Extension for the searches to be usable.

Some marketplace connectors previously developed for WooCommerce needed to be refactored to work with Shopware 6 seamlessly.

Due to a tight timeline, we needed to fix problems that we found in Shopware or in purchased Plugins by ourselves. Sometimes these were errors or customisations and adjustments we wanted. It was a process to learn how to extend, decorate and manage dependencies in a way that even customised packages could be upgraded with Composer.

We needed to relaunch all main sites at once since managing multiple connections to the Warehouse provider would have been a nightmare – and would have required an additional middleware.

The requirements towards server infrastructure, technology stack and deployments are more complex and thus need more attention. The same goes for the development and testing of new features and plugins. For this I created rock solid provisioning, build and deployment pipelines using Github Actions, and Ansible playbooks. New server nodes can be added to the cluster in minutes.

I implemented, extended and customised new marketing tools such as Klaviyo.

I developed many Shopware Plugins and Plugin customisations. I helped Webdesigners and frontend developers with setting up Shopware 6 locally. I made peer reviews. And reviewed Pull Requests. I ported existing marketplace integrations like Shop Apotheke to Shopware 6 and implemented new ones like Galaxus.

Conclusion

Insenio is now able to scale its infrastructure more easily. Adding new sales channel and launching into new markets is way smoother and can be achieved faster. Since there’s only one master stock, the risk of accidental oversales is reduced. Since the relaunch with Shopware 6, Insenio has seen record revenues month after month.

Most importantly, the customer experience is much more pleasant, sites loading is always snappy and corporate identity across sites is more consistent. Code is much cleaner and more transparent to developers (WordPress hooks, actions and filters are quite a nightmare for developers).