#btmusings

Change Data Capture is an important database feature for tracking changes at the transactional level. It enables users or services to retrieve a collection of changes to every table. In cases where records are replicated or synced with services outside of the database, CDC is a useful tool for ensuring data is synced, including when database records are deleted. 

In the API worlds CDC is not as well-known. As recent as mid-2018, Salesforce added CDC to its API capability, including a Streaming API where external systems receive push notifications of such change events. There is little or no information on CDC capabilities in any other API I have come across. 

CDC for APIs should consider the following types of record changes: 

Create

Update

Delete

Undelete

A CDC endpoint should also include the following capabilities: 

Filter by Type - API consumer may only watch out for changes to a subset of records in the API

Filter by Date - API consumer may only watch out for changes since last check

Include updated fields / exclude unchanged fields, 

Reference to original API endpiont or record, except when original record is deleted 

In the case when original record is deleted, include last known version of record data. 

A CDC endpoint might look something like this: 

{ ‘header’: { 

    ‘change_type’:’CREATE’,

    ‘entity_type’: ‘CUSTOMER’,

    ‘entity_id’:925215125,

  },

  ‘data’:{

    ‘name’ : ‘Acme Industries’, 

    ‘email’: ‘[email protected]’, 

    ‘updated_at’: ‘2019-05-01T00:00:00′

  }

}

At work, I have started a project to implement dashboard metrics. The project involves pulling real-time performance metrics from client systems and presenting them in a nice dashboard format in the office. 

Given its low cost, portability and no frills technology, Raspberry Pi 2 (raspberrypi.org) seems the the perfect option for the project. With an LCD monitor powering an RPI2 unit via USB, the only tethering required is the monitor’s own power. Each monitor / RPI2 module can be easily moved around the office - only a nearby power outlet is required. 

The project is off to a good start. The RPI2 unit is running Ubuntu Mate, a light Ubuntu distribution with an X11 interface and pre-installed Firefox browser. 

I am using Geckoboard dashboards (www.geckoboard.com). Due to Chromium compatibility issues, Geckoboard warns against using RPI2 for its dashboards, This has in fact turned out to be true, since even the recent Chromium version 41 fails to display Geckoboard properly. However the Ubuntu Firefox browser works well. RPI2 stays. 

Geckoboard supports dashboard loops, which means I can setup multiple dashboards and the display simply loops through them at set time intervals. There are limitations with controlling the loop - the viewer cannot pause on specific dashboards and resume when done. We will look at custom Javascript to provide such interactivity. 

To effectively manage each dashboard, we may require custom URLs. Each RPI2 browser should point to a unique custom URL that can easily be modified through a code repository. 

We are also looking at incorporating promotional content interwoven between dashboard loops. Such content could span multiple monitors, requiring a TV-wall type implementation. This could simply be implemented through an additional RPI2 unit, using software such as PiWall (piwall.co.uk). 

With some custom CSS styling, I have some basic metrics setup as an example. 

Reduce the Number of Docker Volumes

M2′s “vendor” folder houses pre-packaged modules including M2 core. In recent M2 releases it has grown exponentially in size. Excluding “vendor” from Docker’s mapped volumes improves load times significantly (by 10s of seconds). However, this exclusion introduces a new problem - vendor folder is now in isolation from the Docker environment and testing changes must now be done in Docker environment. The workaround here is to prepackage M2 code into the Docker image. This does feel a little like a hack. 

Excluding vendor code brings page load times to a respectable 5-15 seconds - acceptable for testing, but not production standards. 

Running Grunt Locally 

Grunt, the trendy JS compiler tool, comes pre-packaged in more recent versions of M2. Grunt proves to be useful for M2 because it relieves the user from having to recompile LESS files for testing. In particular the “grunt watch” command automatically regenerates required CSS files from multiple location sources. 

When mapped volumes are involved in the regeneration process, we found “grunt watch” commands to take in excess of 40 seconds to complete! This is not practical for implementing CSS changes. 

My workaround is to rely on the local environment to run grunt watch, which still generates the requires filed in the right (mapped) folders, but without using Docker NGINX server. This reduces compilation times by factors of 3 to 4, to around 8-12 seconds.

Another problem arises, however, with the percona database container we use for the M2 database. Grunt requires a connection to the database for theme path, and databaes connection details are stored in Magento’s app/etc/env,php file. The clincher is this - connection host to the percona database is different when in local environment (”127.0.0.1″) and when in Docker environment (same as [container_name]). 

To get around this, I unmapped the “app/etc/” folder so I have 1 copy of env.php for local, and 1 copy for Docker. The added bonus of this arrangement: I can now run M2′s bin/magento commands in local as well.

After some tweaking, the volumes in my docker-compose.yaml look something like this: 

.....

 web:      # Container     image:    container_name:      # Container name    ports:      - "80:80"      - "8888:8888"      - "8080:8080"      - "8005:8005"    volumes:      - ./htdocs/app/code/:/var/www/app/code/          # Code folder      - ./htdocs/app/design/:/var/www/app/design/     # Theme folder      - ./htdocs/dev/:/var/www/dev/                             # Dev tools inc grunt      - ./htdocs/pub/:/var/www/pub/                            # Media & static content

.....