Getting Started
Creating a Tenant is your first step in Loome Integrate, followed by installing an Agent. You can then create a project, job and task. These steps are all created once and you can then re-use or re-run jobs on-demand or on a schedule.
The components of Loome Integrate
Projects, Jobs and Tasks
Loome Integrate consists of Projects, which contains many Jobs. These Jobs consist of a series of Tasks, which could be any number of the below:
- Execution of a Data Migration
- Execution of a Stored Procedure
- Run a SQL Statement
- Run an OS Command Script
- Run a Python Script
- Run a PowerShell Core script
- And more
Projects might belong to separate departments within your business - for example, Accounting, Marketing, Development etc.
Jobs can be named after the things you want to achieve within each Project, and the tasks within Jobs are the technical steps to achieve this goal. Tasks can belong to multiple Jobs, and each can be configured to be dependant on another if necessary. For example, If Task A fails from within Job A, then Job B will not run. You cannot however, run a single Task on its own, all Tasks are run within their parent Job. Jobs can be scheduled to occur in an automated fashion.
The Overall Process
| Step | Description | Related Documentation |
|---|---|---|
| Step 1: Setup an Agent | An agent will be used to connect to your resources. | Here is a link to learn more about the agent. |
| Step 2: Add users (Optional) | These roles will give permissions to your selected users. You can select which users can create projects and which can see read-only resources. | Find more information about user application roles here. |
| Step 3: Create a Project | A project will contain any jobs, project connections and roles. | Read more about projects. |
| Step 4: Create a Connection | Connections are used to connect to your sources and targets. | Learn more about connections here. |
| Step 5: Create a Job | This will be where you can sequence multiple tasks. | Learn about jobs in Loome here. |
| Step 6: Create a Task | There are various task types that perform different actions, and are contained in jobs and run by agents. | Learn more about tasks here. |
The following sections will show you how to complete each step needed to start running jobs.
This guide will follow steps that require the Administrator role. See our application roles page to learn which roles have permission to take the following actions.
Install an Agent
The agent is the “worker” in Loome Integrate that contains all the code that allows jobs to run. The Agent runs cross platform, meaning that users can leverage data from sources running on both Windows and Linux. For example, Loome Integrate provides Apache Hadoop as a connection type using the Agent’s Linux Support. The Agent should be installed on a host that has access to your data sources, such as a server, for job and task executions from Loome Integrate in the cloud.
The Agent installer is a single Powershell Script that will download the necessary run times, the newest version of the agent and it will also pre-configure your agent to connect to this tenant.
On the Agents page, select Setup Agent.
You can then follow the prompts in the installation wizard for your chosen platform.
Click on ‘Next Steps …’ for your platform.
The install script will automatically be installed as a Local System account. To change the account settings, you will need to modify the service.
Copy and run the command with administrator privileges, which you can copy from Loome once you have chosen a platform, into your Administrator PowerShell Window.
Learn more installing an agent and more information about other platforms here.
Add Users
Administrators can invite additional users to their Loome Integrate Tenant.
Click on Application Roles from the dropdown at the top-right of the screen.
Choose an application role from the tabs at the top of the table.
Then select ‘Add Members’.
This will expand a slideout where you can choose the type of member you would like to add.
Choose whether you want to add a user, a group or an API Key.
To add a user or group, you can search for a user or group by name. (To add a user manually, you can learn more here.)
Click on your chosen user or group. They will then be displayed beneath the dropdown.
You can also manually add an email address and select it as the first option from the dropdown.
Click on Submit to save the chosen user(s).
Create a Project
From the Loome Integrate homepage, you can create a project by selecting Manage Projects on the left-hand slideout or you can select Projects from the menu in the top-right dropdown.
On the ‘Manage Projects’ page, click on New Project at the top-right of the page.
Enter your chosen project name.
Click Next.
Then you can add users to project roles. This will invite additional members from your organization to this project.
Choose the project role from the tabs at the top of this section.
Click on Add Members.
Choose to add a user, group or API Key.
To add a user or group, type in their name in the search field.
Select the user or group from the dropdown.
Click on Submit.
Click Create at the bottom-right and this new project will appear in the project list.
Add a Connection
Connections in Loome Integrate are your data sources - where you can pull data from - for migration tasks. These can be on-premises, such as a SQL server, or any number of cloud-based connections.
Loome Integrate also allows for the migration of data to and from file-based data stores. For example, Loome Integrate treats flat files like any other connection - each file in a directory is essentially a table in a database. Loome Integrate also supports both local files and Azure blob. Loome Integrate is also able to leverage the power of its flat file processor to support Azure Blob Storage Containers as a source and target connection.
This guide will go through creating a connection. You can learn more about using file definitions and cluster definitions as a source and target in our full guide.
Click on Connections from within the dropdown menu in the top-right corner.
Once on the Connections page, click Add Connection.
Choose your data sources type. You can filter this list by typing the connection name you are looking for in the filter field. In this example, we will be connecting to a Microsoft SQL Server database.
Then choose if this connection will be available to all projects or only to selected projects.
If you choose selected projects, you can then choose from a list of all projects in this tenant. This connection will only be available in these projects and will not be displayed when creating tasks in other projects.
After you have chosen your projects, click Next.
Provide a Connection String.
Loome Integrate has connection templates automatically generated for your source type, and you can choose one from the dropdown below the connection string.
Then complete the template by replacing fields with any required information.
For example, for a Microsoft SQL Server connection, the following standard template only requires your server address and your database name, as shown below.
Depending on the connector type, you may need to provide Loome Integrate your credentials in order to validate your connection.
Provide your credentials in the user name and password fields.
Click Submit to save this connection.
Learn more about adding connections here.
Create a Job
To add a new job, click on your project, and then click New Job in the top-right corner.
You can then provide a Job Name and click Submit to create a simple job with default customization.
If you would like to modify the job to your requirements, you can change the following fields to customize the behavior of your job.
Provide a Job Name to identify your job.
Choose a Logging Level. The Logging Level refers to how detailed the execution logs will be after running a Job.
The Logging Batch Size determines the rate at which logs are submitted to the Loome Integrate Online API from the Agent. By default, the agent will submit logs in batches of 5. If there is a job that creates a large amount of logs (>100 pages worth) it’s advised that this value is increased so that less load is put on your tenant.
(Please note that increasing the batch size will slow down the rate at which realtime logging occurs in the job execution screen, however this does not slowdown the performance of the job execution itself.)
Choose to be emailed after a job is run by selecting an Email Notification Rule. It will enable job driven email notifications on success or failure of a job.
If you are using an email notification rule, provide Email Recipients.
Enter a comma separated list of recipients who will receive the email notifications.
Choose to enable a Retry Count by providing a number to this field. The Retry Count determines the number of times each task will be repeated if it fails on its first execution. It will repeat the task the amount of times you have chosen until it completes successfully or reaches the job’s retry count in failures.
After you have modified your chosen fields, click Submit at the bottom-right to save this job.
Create a Task
Click on ‘Add Task’ from within your selected job to create a new task.
Choose a Task Name.
Select an Agent. This can be the one we added above.
Choose your Task Type, in this example it will be a Data Migration. Learn about other task types here.
Next we need to choose where the source data will be imported from and the target location we will copy into. Choosing your Source Connection and Schema will define which tables you are able to select for migration in the next step.
Select your Source Connection from the dropdown.
Then choose the Source Schema where the data is located from the dropdown.
Choose a Target Connection from the dropdown.
Then choose a Target Schema or you can create a new schema by selecting the option below and typing in the new schema name. This will be the target location your data will be copied to.
This next step displays the tables within the schema that you chose on the last screen.
Use the plus button + on the right of the screen to select the tables you wish to migrate from the Source Connection.
Tip: If you change your mind, removing a table is as easy as clicking the minus button on the right side of the ‘Target’ section.
Once you are happy with the tables you’ve selected, click ‘Submit’ to save the task. You can edit this saved task at any time using the edit button on the Job screen.
Your task is now created and the next step is to run this job. Your Data Migration task will now appear in your ‘Sequenced Tasks’ list, and you will be able to run the task as part of the job at anytime with the ‘Run Now’ button - located in the top right. (You can also create a schedule to run this job at your chosen time.)
Be aware that this will run all Sequenced Tasks as part of the job.
Checking the ‘Show Logs after Execution’ box will take you to a live job execution page.
Loome Integrate will show the logs and status of your job in real time - updating the status icon to a green tick on completion, or a red flame on failure.
