FTP component

The FTP component is able to connect to FTP servers (FTP, SFTP, FTPS) and read or write files. It can act both as inbound component (read files from the server), and outbound component (write files to the server).

Basic configuration

The FTP component has the following basic configuration options:

• Force inbound
• Protocol
• Host
• Port
• Username
• Password
• Private key
• Path
• File name
• File type
• Charset
• File exists
• Delay

Force inbound

Description

Settings this to Yes will force the component to turn from outbound to inbound and consume a file from the server. This has the same behavior as putting the component below an Enrich component that is set to Override. This means that the current body will be overridden with the new body of the consumed file.

Remarks

• Only available when the component is outbound.
• Only one file will be consumed when the component is forced to be inbound.
• This options only becomes visible when another component is connected on the left side of this component.
• You can set this option to Yes when this component is the first component at the bottom of an Enrich component to show the inbound options.

Protocol

Options

• SFTP (default)
• FTPS
• FTP

Description

Specify whether the server is FTP, SFTP or FTPS.

Host

Description

URL to the FTP server, e.g. ftp.example.org or IP address, e.g. 149.210.209.32.

Port

Description

Specify the FTP port if the server uses non-default ports. When left empty, defaults are FTP: 21, SFTP: 22.

Username

Description

Username used to log in on the FTP server.

Password

Description

Password used to log in on the FTP server.

Private Key

Description

Either specify the location of an SSH private key file (e.g. uploaded with the File component) or directly upload a file using the upload button on the right.

Remarks

• Only available when the Protocol is set to SFTP.
• The private key file needs to be of the format OpenSSH.
• The uploaded key is identical for all environments.

Path

Description

Path to the target directory. In case of inbound this is the directory where Dovetail will be picking up files from, if used as outbound this is the directory where Dovetail will write files to.

Remarks

• If you include a / at the end of the path, the component will try to copy the whole file path and not the file!

File Name

Description

Get specific file with this name when inbound or set file name when outbound.

File Type

Options

• Text (default)
• Binary

Description

Specify the file(s) type.

Charset

Description

Specifies the encoding of the file.

Remarks

• Only available when the component is outbound and the file type is set to text
• When processing large text files see Producing large text files below.

File Exist

Options

• Override (default)
• Append
• Ignore
• Fail

Description

Specify what to do if a file with the same name already exists.

Remarks

• Only available when the component is outbound.

Delay

Description

Milliseconds to wait before the next poll of the file/directory.

Remarks

• Only available when the component is inbound.

Advanced configuration

The FTP component has the following advanced configuration options:

• Passive mode
• Disconnect
• Auto create
• Order by
• Maximum batch size
• Recursive
• Delete file
• Move to
• Move failed to
• Read lock
• Include
• Exclude
• Initial Delay

Passive Mode

Options

• Yes (default)
• No

Description

Specify whether FTP should operate in active or passive mode.

Disconnect

Options

• Yes (default)
• No

Description

Whether or not to disconnect from the FTP server right after use. It will reconnect when the component is used again.

Remarks

• Only available when the component is inbound.

Auto Create

Options

• Yes (default)
• No

Description

Automatically creates missing directories in the file path.

Remarks

• Only available when the component is outbound.

Order by

Options

• Default (no order guaranteed)
• File name (ascending)
• File name (descending
• Modified date (ascending)
• Modified date (descending)

Description

Specify the order in which files shall be consumed.

Remarks

• Only available when the component is inbound.

Maximum batch size

Description

Maximum number of files processed on each poll. This is empty by default which means all present files will be processed at once.

Remarks

• Only available when the component is inbound.

Recursive

Options

• Yes
• No (default)

Description

When inbound, specifies whether Dovetail should process files only in the specified path (No) or in all subdirectories of the path too (Yes).

Remark

When set to Yes you must make sure that your exclude pattern will ignore the directory were you place the processed files, otherwise you will end up with a infinite loop.

Delete file

Options

• Yes
• No (default)

Description

If set to Yes, the file will be deleted after it is processed.

Remarks

• Only available when the component is inbound.
• When set to Yes, the Move option is ignored and hidden.

Move to

Description

Relative path to folder where files should be moved to when done processing.

Create and move files into this folder when done processing. This should be a relative path. In order to move to higher paths use ../[folder name] or ../../[folder name].

Remarks

• Only available when the component is inbound.
• Only used and shown when Delete file is set to No.
• The default behavior is to create a directory named .dovetail and move everything into that directory when files are processed. However some FTP servers don't allow creating .\* files or directories. In this case, you need to change the name of the .dovetail directory to something that is accepted by the server.

Move failed to

Description

Relative path to folder where files should be moved to when they fail processing.

Move files into this folder when they fail processing. This should be a relative path. In order to move to higher paths use ../[folder name] or ../../[folder name].

Remarks

• Only available when the component is inbound.
• The default behavior is to create a directory named .error and move files into that directory when they fail to process. However some FTP servers don't allow creating .\* files or directories. In this case, you need to change the name of the .error directory to something that is accepted by the server.

Read lock

Options

• none
• changed

Description

If set to changed, only consumes the files if it has exclusive read-lock on the file (i.e. the file is not in-progress or being written). It will wait until the file lock is granted.

Remarks

• Only available when the component is inbound.

When set to changed:

• It requires write permissions on the FTP server.
• It uses file length or file modification timestamp to detect whether the file is currently being changed or not. It will at least use 1 second to check this and there will also be some time between each check, so consuming files will be slower.
• It will require a lock on the latest file until a new file has been written to the FTP server. It will not be processed until a new file has been dropped at the ftp server.

Include

Description

Used to include files, if the filename matches the regular expressions pattern.

Remarks

• Only available when the component is inbound.

Exclude

Description

Used to exclude files, if the filename matches the regular expressions pattern.

Remarks

• Only available when the component is inbound.

Initial delay

Description

Milliseconds to wait before polling the file/directory starts.

Remarks

• Only available when the component is inbound.

Producing large text files (> 10 MB)

When producing large text files like XML or JSON might be good to set files to binary which means it does not have to read all the file's content into memory to validate the charset. This is better performance-wise and can prevent errors parsing the file later on in the flow.

Consuming folders with a lot of files

When the FTP component acts as a inbound component it can't pick up multiple files at the same times and process them. Every time the FTP Component pols the server it will create a list of all the available files and pick one to process. For this reason consuming a folder with a large amount of files can take more time than expected.

To improve performance, files can be split up over different subfolders, so that different FTP component can poll each subfolder.

Remarks

• Authentication to SFTP servers can be provided by specifying a password and/or an SSH private key file.
• FTPS supports FTP over TLS up to version 1.2.
• The variables ${header.<headername>} and ${headers.<headername>} can be used for all properties of the FTP component.
• Order by and Maximum batch size can be combined to achieve specific behaviours such as FIFO if you set Order by: Upload date (descending) and Maximum batch size: 1.