- DarkLight
API v1 - Git Integration
- DarkLight
Overview
This article is a guide to using the source control management (SCM) API in Matillion ETL. Matillion ETL uses Git for its version control, allowing teams to track changes (both small and large) to a project or a shared job with speed and efficiency.
Note
- Consider whether Git integration is appropriate for your Matillion ETL practices with our When to Choose Git article.
- Git integration
- Git FAQ
- If you are new to Matillion ETL's API, read our Matillion API introduction.
- Explore the v1 API map.
- To jump to the v1 API, just run {host:port}/rest/v1 in your browser.
- Git integration in Matillion ETL is an Enterprise Mode feature. Read about the benefits of upgrading your license to Enterprise Mode here.
- All information in <angle brackets> should be replaced with information based on your own configuration.
- The API has a different set of endpoints for working with (a) projects and (b) shared jobs. The examples in this article are for use with projects, with the endpoints needed for use with shared jobs listed in the table at the end of the article.
Initialize A Local Repository
To initialize a local repo using the Matillion v1 API, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/init
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Project/project/name/Docs/version/name/default/scm/init
The variables in the above example are listed immediately below for added clarity. For security reasons, the instance address remains hidden.
- groupName = Docs_Project
- projectName = Docs
- versionName = default
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.
Method | Body (JSON) | Description |
---|---|---|
POST | { "commitMessage": "<Your Commit Message>", "userIdentifier": { "username": "<username>", "emailAddress": "<emailaddress>" } } | Initializes a new repository in this Matillion ETL version. This request creates an initial Git commit within Matillion ETL.
|
Clone Remote Repository
To clone a remote repository, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/clone
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/clone
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.
Method | Body (JSON) | Description |
---|---|---|
POST | { "remoteURI":"<URL>", "auth": { "authType": "HTTPS", "username": "<username>", "password": "<password>" } }or: { "remoteURI":"<URL>", "auth": { "authType": "SSH", "privateKey": "<key>", "passphrase": "<passphrase>" } } | Clone an existing remote repository in this Matillion ETL version. Successful requests will initialise from or clone your remote repository.
|
Commit to Local Repository
To commit changes to a local repo using the Matillion v1 API, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/commit
Here is an example of this request with example variables added (for security, the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Project/project/name/Docs/version/name/default/scm/commit
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.
Method | Body (JSON) | Description |
---|---|---|
POST | { "branchName":"<branch>", "changes":[{"type":"ADDED","filePath":"<path>"}], "commitMetadata":{"commitMessage":"<Your Commit Message>","userIdentifier":{"username":"<username>","emailAddress":"<email address>"}} } | Commit changes to a shared repository.
|
Create Branch
To create a new branch on the current commit, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/createBranch
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
{ "httpMethod": "POST", "name": "Create new branch", "path": "/createBranch", "arguments": [ { "style": "DATA", "name": "createBranchInfo", "type": "CreateBranchInfo" } ], "type": "String", "example": "/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/createBranch WITH POST DATA", "totalPath": "/group/name/{groupName}/project/name/{projectName}/version/name/{versionName}/scm/createBranch" }
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
POST | { "branchName": "<name of new branch>" } | Create a new branch on the current commit. The POST call will return the branch name along with the commit ID.
|
Switch Commit
To switch to another Git commit using the API, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/switchCommit
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/switchCommit
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
POST | { "commitID": "<ID>" } | Switch to an existing commit by passing the commit ID. You can find commit IDs by using the 'getState' endpoint detailed in the 'Get Commits' section of this article.
|
Server Response
A successful POST call to the endpoint will return the following response from the server:
{ "success": true, "result": "Success" }
A failed call will return a brief error description, for example:
{ "success": false, "result": "Error while running command: An error occurred while switching commits\nCaused by: ab01418adceb9a6ec51e49ee7d7787dde74220e" }
Fetch from Remote Repository
To perform a fetch from a remote repository, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/fetch
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/fetch
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), a description of the endpoint, and any pertinent notes that users should consider.
Method | Body (JSON) | Description |
---|---|---|
POST | { "auth": { "authType": "HTTPS", "username": "<username>", "password": "<password>" }, "fetchOptions": { "removeDeletedRefs": "<true/false>", "thinFetch": "<true/false>" } }or { "auth": { "authType": "SSH", "privateKey": "<key>", "passphrase": "<passphrase>" }, "fetchOptions": { "removeDeletedRefs": "<true/false>", "thinFetch": "<true/false>" } } | Fetch from a remote repository. A remote repository must be configured before fetch requests can be made.
|
Push to a Remote Repository
To push to a remote repository, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/push
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/push
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
POST | { "auth": { "authType": "HTTPS", "username": "<username>", "password": "<password>" }, "pushOptions": { "atomic": "<true/false>", "forcePush": "<true/false>", "thinPush": "<true/false>" } }or { "auth": { "authType": "SSH", "privateKey": "<key>", "passphrase": "<passphrase>" }, "pushOptions": { "atomic": "<true/false>", "forcePush": "<true/false>", "thinPush": "<true/false>" } } | Push to a remote repository.
|
Configure a Remote Repository
To configure a remote repository, make the following POST call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/remote
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
POST | { "url": "{your_remote_uri}" } | Configure a remote repository.
|
Get Remote (returns URL)
To return the URL of your remote repository, make the following GET call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/remote
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/remote
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
Get Commits
To retrieve a list of each Git commit on the local repository, make the following GET call:
<instance address>/rest/v1/group/name/<groupName>/project/name/<projectName>/scm/getState
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/scm/getState
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
GET | N/A | Provides a list of commits on the local repository. |
Get Differences
To get differences between the current Matillion version and the branch head, make the following GET call:
<instanceaddress>/rest/v1/group/name/<groupName>/project/name/<projectName>/version/name/<versionName>/scm/differences
Here is an example of this request with variables added (the instance address has not been added):
{host:port}/rest/v1/group/name/Docs_Proj/project/name/Docs/version/name/default/scm/differences
Here is the endpoint as seen when running the {host:port}/rest/v1 in a browser:
The table below documents the type of API endpoint, the body (in a JSON format), and a description of the endpoint.
Method | Body (JSON) | Description |
---|---|---|
GET | N/A | Returns the difference between the current version (including uncommitted changes) and the head. |
The following is an example of the server response to the GET call:
{ "success": true, "result": { "branches": [ "master" ], "changes": { "dotfileChanges": [], "nonDotfileChanges": [ { "type": "MODIFIED", "filePath": "ROOT/o1.ORCHESTRATION" }, { "type": "MODIFIED", "filePath": "version" } ] } } }
The type response can be ADDED, REMOVED, or MODIFIED.
Shared Jobs
Many of the functions listed above can also be used with shared jobs, by making a POST call with the form:
<instanceAddress>/rest/v1/git/shared-jobs/<endpoint>
The endpoints which can be used for shared jobs are listed in the following tables.
Function | Method | Endpoint | POST Body |
---|---|---|---|
Initialize shared jobs repository | POST | <instanceAddress>/rest/v1/git/shared-jobs/init | See Initialize A Local Repository |
Clone remote repository for shared jobs | POST | <instanceAddress>/rest/v1/git/shared-jobs/clone | See Clone Remote Repository |
Commit shared job changes to local repository | POST | <instanceAddress>/rest/v1/git/shared-jobs/commit | See Commit to Local Repository |
Switch to another shared job commit | POST | <instanceAddress>/rest/v1/git/shared-jobs/switchCommit | See Switch Commit |
Fetch shared job from remote repository | POST | <instanceAddress>/rest/v1/git/shared-jobs/fetch | See Fetch from Local Repository |
Push shared job to remote repository | POST | <instanceAddress>/rest/v1/git/shared-jobs/push | See Push to a Remote Repository |
Get differences between the current Shared Job version and the branch head | GET | <instanceAddress>/rest/v1/git/shared-jobs/differences | N/A |
Set remote url for git shared jobs repository | POST | <instanceAddress>/rest/v1/git/shared-jobs/remote | { "url": "<URL>" } |
Create a new branch on current commit | POST | <instanceAddress>/rest/v1/git/shared-jobs/branch | { "branchName": "<name of new branch>" } |
Return the URL of the remote repository | GET | <instanceaddress>/rest/v1/git/shared-jobs/remote | N/A |
Return the history of commits in all branches of the repository | GET | <instanceaddress>/rest/v1/git/shared-jobs/state/<max_count>/<skip>
| N/A |
Delete a branch | DELETE | <instanceaddress>/rest/v1/git/shared-jobs/branch/<branchName>
| N/A |
Delete the repository | DELETE | <instanceaddress>/rest/v1/git/shared-jobs | N/A |
Merge a branch, assuming there are no conflicts. If conflict occurs, the server returns a list of conflicts. If the request contains conflict resolutions in the conflictsResolution field of the POST Body, then the resolutions are applied. | POST | <instanceaddress>/rest/v1/git/shared-jobs/merge | See below |
The /merge endpoint has the following POST Body:
{ "commitMetadata": { "commitMessage": "<message>", "userIdentifier": { "username": "<name>", "emailAddress": "<email>" } }, "checkoutAfterMerge": "<true | false>", "mergeFrom": "<source branch>", "mergeInTo": "<target branch>", "conflictsResolution": [ { "package":"<package>", "name": "<name>", "revision": <number>, "choice" : "<OURS | THEIRS>" } ] }
Contact Us
If any help is needed navigating the Git API in Matillion ETL, please read Getting Support.