Installation

Requirements

The installation of Fast2 requires a few environment specifications to run properly :

  • JRE11 : you can either get it from Oracle or OpenJDK. If you have multiple JDK/JRE already installed, point out the correct one in the config/env.properties file of the Fast2 installation folder.
  • RAM : we highly recommend having 8GB or more, although 4GB is the strict minimum.
  • Processor : 8 CPUs.

Make sure to confirm the compatibility between Elasticsearch and your environment at Elasticsearch Support Matrix.

Fast2 packages

The Fast2 distribution you need depends on your target environment. It exists three way to deploy a Fast2 :

  • locally : Classic package, its an all-in-one zip files
  • AWS : Standard AMIs
  • K8S : Docker Images

Each distribution ships the following

  • A broker with an embedded worker and a user interface
  • An additional worker with all tasks catalog
  • A template to create workers with custom tasks

Root folder anatomy

Root folder anatomy

Start Fast2 Broker

Whether Fast2 is launched from the batch file or as a service on your environment, the UI will be available at http://localhost:1789/

By default, Fast2 Broker starts an embedded Elastic Search and an embedded Fast2 Worker.

All commands below are to be run under the Fast2 install path (where the Zip has been unzipped).

From command line

Go into the Fast2 install folder, and run :

C:\path-to-fast2\> startup-broker.bat

Administrator rights might be required since Fast2 will handle some port communications.

To end the Fast2 process, just hit Ctrl+C in the command line the startup file opened.

The following Linux installation steps work for most of Unix-based sytems. Elasticsearch cannot be started from root user, you will need to create a secondary user to start the database binary alongside your Fast2 process.

Once connected as the latter user, start Elasticsearch via its binaries.

Then, since you started Elasticsearch manually, disable the command triggering Fast2 to start the embedded Elasticsearch, from the config/application.properties file :

broker.elasticsearch.embedded.enabled=false

You can now properly execute the following script:

$ ./startup-broker.sh

To quit the Fast2 process, just hit Ctrl+C in the command line the startup file opened.

As service

Go into the Fast2 installation folder, and open the Windows Command Prompt.

To install the service :

C:\path-to-fast2\service\windows> Fast2_broker_service.exe install

Your machine may prompt a message asking to download .NET components. Please click [OK] and proceed.

Once this command is complete, you should see in your services registory a newly installed Fast2 service. You can start/stop/restart it as any other service, or via the Command Prompt (just replace install in the previous command by start/stop/uninstall/restart/status).

The logs of the service will be available from the path-to-fast2\service\log folder.

There are several ways to create a service under linux distribution. We will do it through systemd. Its major benefit is that it has been the default init system for the majority of linux distributions (Ubuntu, Red Hat, Fedora…).

Execution path

Edit the ExectStart field from the file service/linux/fast2-broker.service by changing the PATH/TO/FAST2 portion: set it to Fast2 install path.

[Unit]
Description=Fast2 Broker

[Service]
ExecStart=/home/userName/fast2-complete-package-2.0.0/startup-broker.sh

[Install]
WantedBy=default.target

Now link it to the /etc/systemd/system directory through a symbolic link.

$ sudo ln -s SOURCE TARGET

$ sudo ln -s service/linux/fast2-broker.service /etc/systemd/system

If the links are broken once they’re created, you probably need to put an absolute path for the target as follow ;

$ sudo ln -s /home/userName/fast2-complete-package-2.0.0/service/linux/fast2-broker.service /etc/systemd/system

Next, reload systemd services unit and enable them :

$ systemctl daemon-reload
$ systemctl enable fast2-broker.service

The terminal should prompt the following message :

Created symlink from /etc/systemd/system/default.target.wants/fast2-broker.service to /etc/systemd/system/fast2-broker.service.
Script uses

Test your script by starting it and then checking the status :

$ service fast2-broker start

$ service fast2-broker status

OR

$ systemctl start fast2-broker.service

$ systemctl status fast2-broker.service

You can restart or stop the service at anytime with the commands :

$ service fast2 start | restart | stop | status

Start Fast2 Worker

The Broker starts an embedded worker by default. If you wish to start multiple workers, just hit :

C:\path-to-fast2\> startup-worker.bat
./startup-worker.sh

If the worker and broker are not booted up on the same machine, you need to setup the Broker host name in the worker configuration file. Edit the file config/application.properties and modify broker.host accordingly.

You can setup Fast2 Worker as a service the same way you did for the Fast2 Broker.

Elasticsearch setup

Every object passing through Fast2 is stored into Elasticsearch. Metadata carried by the document are also kept in the warehouse. The major benefit is the opportunity to check whether everything is going well by making counters about documents/data processed during your migrations. In addition, we can easily come back in case of server crashes. Nothing will be lost and Fast2 will precisely know where it all stopped. There is the logic behind real-time backups in ES.

Index names

Each Elasticsearch index referenced by Fast2 will be written with a f2_ prefix. An index is always written in lower case even if the campaign name in Fast2 contains characters in upper cases.

At the first broker run, some indexes will be automatically created :

  • f2_campaigns: list of existing campaigns with processing dates and status
  • f2_campaigns_sources: links between campaigns and workers having performed this campaign
  • f2_queue_settings: reference information about source and task threads
  • f2_jobs_settings: gather the configuration of save jobs
  • f2_jobs_info: information about jobs past execution details

For each new campaign of Fast2, an index will be created: if you decided to run a new campaign named EcmInjection, the new index will be f2_ecminjection_try1.

Elasticsearch without Kibana

To visualize data stored in Elasticsearch without using Kibana, we recommand to use web navigator plugins. These GUI are focused on table visualisation of your documents, more easily than with Kibana. You can browse your data as classic SQL tables. The downside is ElasticVue does not offer any metrics, graphs or anaytics options. Two different plugins exist, depending on your web navigator:

Troubleshooting

Linux permissions

The elasticsearch and kibana binaries files are not allowed to being executed by default. Open a terminal, go to your Fast2 root folder and execute the following command:

sudo chmod -R 777