Overview

This page covers the technical details of how to upgrade and rollback between the Matillion ETL non-universal and universal packages along with an overview of the reasoning behind the project.

Background: Consolidation of .rpm packages

Prior to v1.60, Matillion ETL was published as three yum packages:

matillion-emerald
matillion-emerald-byol
matillion-emerald-cdata

Each installation of Matillion ETL would then have one combination of:

matillion-emerald AND matillion-emerald-cdata

Or:

matillion-emerald-byol AND matillion-emerald-cdata

Both of the matillion-emerald and matillion-emerald-byol packages then had both cloud platform and cloud data warehouse specific builds. For example:

emerald-1.46.10-centos7-aws-snowflake-platform.rpm   (matillion-emerald package)
emerald-1.59.9-centos7-azure-snowflake-platform.rpm  (matillion-emerald package)
emerald-1.51.9-centos7-aws-redshift-byol.rpm (matillion-emerald-byol package)

However, matillion-emerald-cdata was always published as a single universal build. For example:

emerald-cdata-1.50.12.rpm (matillion-emerald-cdata package)
emerald-cdata-1.56.10.rpm (matillion-emerald-cdata package)

As will be obvious, this large amount of builds/variation resulted in quite a complicated installation and complicated the tasks of building, validating, distributing, and providing support.

What's new: matillion-universal package

Therefore, as part of the v1.60 build, there have been two important changes:

1. The matillion-emerald and matillion-emerald-byol packages have been consolidated to form the matillion-universal package.
2. There are no longer cloud platform and cloud data warehouse specific builds for the matillion-universal package. Instead, there is one universally compatible build per Matillion Version.

For example:

emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)
emerald-metl-1.60.2-<unique id>.rpm   (matillion-universal package)

From v1.60 onwards, each Matillion ETL installation will require:

matillion-universal *AND* matillion-emerald-cdata

Finding the matillion-universal packages

Two new yum repositories have been created, which can be found at the following URLs:

1. matillion-universal ——— https://artifacts.matillion.com/rpm/matillion-metl/stable/
2. matillion-emerald-cdata ——— https://artifacts.matillion.com/rpm/matillion-cdata/stable/

An example yum repo config for each is shown below:

Usually found at /etc/yum.repos.d/matillion-cdata.repo:

[matillion-cdata]
name=matillion-cdata
baseurl=https://artifacts.matillion.com/rpm/matillion-cdata/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-cdata/stable/matillion.key
gpgcheck=1
enabled=1

Usually found at /etc/yum.repos.d/matillion-universal.repo:

[matillion-universal]
name=matillion-universal
baseurl=https://artifacts.matillion.com/rpm/matillion-metl/stable/
gpgkey=https://artifacts.matillion.com/rpm/matillion-metl/stable/matillion.key
gpgcheck=1
enabled=1

Upgrading to the universal packages

Users have two available methods to upgrade to the universal packages (Matillion ETL v1.60 and beyond).

Method 1: Updating/Upgrading via migration

To upgrade via migration, please launch a machine image of version 1.60 or higher and follow the steps here: Updating: Machine Images & Migration .

Method 2: Updating/Upgrading in-place (non-clustered environments)

Updating via the GUI is possible for non-clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package.

To do this, please follow the steps here: Updating: In-place updates.

Updating/Upgrading in-place (clustered environments)

Updating via the console is possible for clustered environments. This will transfer the existing installation from matillion-emerald or matillion-emerald-byol onto the matillion-universal package in place.

1. Ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:

service tomcat stop

2. To identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:

yum list installed | grep matillion-emerald

3. This will return either:

matillion-emerald-byol-<some version>

Or:

matillion-emerald-<some version>

4. To "swap" to the universal packages, the following command can be run:

yum swap <package from above> matillion-universal

5. For example, if swapping matillion-emerald-byol:

yum swap matillion-emerald-byol matillion-universal
Loaded plugins: fastestmirror, matillion-ensure-yumhook

Loading mirror speeds from cached hostfile

base: download.cf.centos.org
epel: d2lzkl7pfhq30w.cloudfront.net
extras: download.cf.centos.org
updates: download.cf.centos.org

Resolving Dependencies

--> Running transaction check

---> Package matillion-emerald-byol.noarch 0:1.59.12-1 will be erased

---> Package matillion-universal.noarch 0:1.60.2-1 will be installed

--> Processing Dependency: matillion-emerald-cdata = 1.60.2 for package: matillion-universal-1.60.2-1.noarch

--> Running transaction check

---> Package matillion-emerald-cdata.noarch 0:1.59.12-2 will be updated

---> Package matillion-emerald-cdata.noarch 0:1.60.2-1 will be an update

--> Finished Dependency Resolution

Dependencies Resolved
==================================================================================================================================

PackageArch  VersionRepository  Size
Installing:

matillion-universalnoarch1.60.2-1   matillion-universal351 M

Removing:

matillion-emerald-byol noarch1.59.12-1  @matillion 431 M

Updating for dependencies:

matillion-emerald-cdatanoarch1.60.2-1   matillion-cdata457 M
Transaction Summary
Install  1 Package

Upgrade ( 1 Dependent package)

Remove   1 Package
Total download size: 808 M

Is this ok [y/d/N]: y

Downloading packages:

...

Excluded for brevity

...  4/4
Removed:

matillion-emerald-byol.noarch 0:1.59.11-1
Installed:

matillion-universal.noarch 0:1.60.2-1
Dependency Updated:

matillion-emerald-cdata.noarch 0:1.60.2-1
Complete!

6. Finally, restart the service once both nodes have been upgraded:

service tomcat start

Once the upgrade is confirmed to be successful, to prevent the old packages being installed alongside matillion-universal please run the command:

mv /etc/yum.repos.d/matillion.repo /etc/yum.repos.d/matillion.repo.bkp

This will de-index the repository file mirror for the old packages.

Ongoing updates on matillion-universal

Updates on 1.60 and beyond are then consumed by running the following command:

yum update matillion-universal

Not by running yum update matillion-*

Rollback

Rolling back via the console is possible for all environments. This will transfer the existing installation from matillion-universal back onto either the matillion-emerald or matillion-emerald-byol.

1. First, ensure Tomcat is not running on all nodes before proceeding. To do this, SSH onto all nodes and as root run:

service tomcat stop

2. If previously the matillion.repo repository file was de-indexed, please re-index by running:

mv /etc/yum.repos.d/matillion.repo.bkp /etc/yum.repos.d/matillion.repo

This will re-index the repository file mirror for the old packages.

3. Now, to identify whether matillion-emerald or matillion-emerald-byol are installed, the following can be run:

yum list available | grep matillion-emerald

This will return either:

matillion-emerald-byol-<some version>

Or:

matillion-emerald-<some version>

4. To "swap" back off the universal packages, the following command can be run:

yum swap matillion-universal <package from above>

5. For example, if swapping for matillion-emerald-byol:

yum swap matillion-universal matillion-emerald-byol

Transaction output:

    yum swap matillion-universal matillion-emerald-byol
Resolving Dependencies
    --> Running transaction check
    ---> Package matillion-emerald-byol.noarch 0:1.59.12-1 will be installed
    ---> Package matillion-universal.noarch 0:1.60.2-1 will be erased
    --> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================
Package   Arch  VersionRepository   Size
==================================================================================================================================
Installing:
matillion-emerald-byolnoarch1.59.12-1  matillion   351 M
Removing:
matillion-universal   noarch1.60.2-1   @matillion-universal 431 M
Transaction Summary ==================================================================================================================================
Install  1 Package
Remove   1 Package
Total download size: 351 M

Is this ok [y/d/N]: y
...

excluded for brevity

...

Removed:

matillion-universal.noarch 0:1.60.2-1
Installed:

matillion-emerald-byol.noarch 0:1.59.12-1
Complete!

6. In the above transaction, the matillion-emerald-cdata package was not pushed down, so that needs to be downgraded to match the version of either the matillion-emerald or matillion-emerald-byol package. Therefore, run the command:

yum downgrade matillion-emerald-cdata-1.59.12

Example output:

yum downgrade matillion-emerald-cdata-1.59.12
Resolving Dependencies

--> Running transaction check

---> Package matillion-emerald-cdata.noarch 0:1.59.12-2 will be a downgrade

---> Package matillion-emerald-cdata.noarch 0:1.60.2-1 will be erased

--> Finished Dependency Resolution
Dependencies Resolved
==================================================================================================================================

Package   Arch Version Repository   Size
Downgrading:

matillion-emerald-cdata   noarch   1.59.12-2   matillion   458 M
Transaction Summary
Downgrade  1 Package
Total download size: 458 M

Is this ok [y/d/N]: y

Downloading packages:

emerald-cdata-1.59.11.rpm  | 458 MB  00:00:18

Running transaction check

Running transaction test

Transaction test succeeded

Running transaction

Installing : matillion-emerald-cdata-1.59.12-2.noarch   1/2

Cleanup: matillion-emerald-cdata-1.60.2-1.noarch2/2

Verifying  : matillion-emerald-cdata-1.59.12-2.noarch   1/2

Verifying  : matillion-emerald-cdata-1.60.2-1.noarch2/2
Removed:

matillion-emerald-cdata.noarch 0:1.60.2-1
Installed:

matillion-emerald-cdata.noarch 0:1.59.12-2
Complete!

7. Finally, restart the service once both nodes have been downgraded:

service tomcat start

Upgrading to a specific release

To list available versions:

yum list matillion-universal --showduplicates

To upgrade to a specific version (don't include < and >):

yum update matillion-universal-<your-chosen-version>

To downgrade to a specific version (don't include < and >):

yum downgrade matillion-universal-<your-chosen-version> matillion-emerald-cdata-<your-chosen-version>