Skip to main content

Loop Amazon S3

Synopsis

This operator loops over all files in the specified bucket/folder from the Amazon S3 cloud storage.

Description

配置您的Amazon S3帐户后,you can process all Amazon S3 files within the selected folder.

Be aware that the operator cannot read the file as example set. For this reason, you must connect the file input in the inner process of this operator to another appropriate operator to process the file. For example, if you want to load Excel files from your Amazon S3 folder, you must connect the file input in the inner process with the Read Excel operator.

Input

in

Optional input data which is delivered to the inner process.

connection

This input port expects a Connection object if any. See the parameterconnection entryfor more information.

Output

out

Output data of the inner process.

connection

This output port delivers the Connection object from the input port. If the input port is not connected the port delivers nothing.

Parameters

Connection source

This parameter indicates how the connection should be specified. It gives you two options, predefined and repository. The parameter is not visible if theconnectioninput port is connected.

Connection entry

This parameter is only available when theconnection sourceparameter is set torepository. This parameter is used to specify a repository location that represents a connection entry. The connection can also be provided using theconnectioninput port.

Connection

This parameter is only available when theconnection sourceparameter is set topredefined. The connection details for the Amazon S3 connection have to be specified. If you have already configured a Amazon S3 connection, you can select it from the drop-down list. If you have not configured a Amazon S3 connection yet, select the icon to the right of the drop-down list. Create a new Amazon S3 connection in the Manage connections box. The access key, secret key and the region are required. Note: It is very important to select the correct region for your connection. Otherwise an error occurs.

Folder

Provide the name of the Amazon S3 'folder' over which you want to loop. Note that the concept of folders does not exist in Amazon S3, so the default delimiter ('/') is used to represent them. If your file was stored as 'name1/name2/my_file.xls' on Amazon S3, the file 'my_file.xls' would be displayed as residing in the folder 'name1/name2/'.

Filter

Optional filter via a regular expression which is used to exclude files from looping over them, e.g. 'a.*b' for all files starting with 'a' and ending with 'b'. Ignored if empty.

Filtered string

Indicates which part of the file name is matched against the filter expression.

  • file_name: Filtered on the name, e.g. 'myfolder/myfile.txt'
  • full_path: Filtered on the full path, e.g. 'mybucket/myfolder/myfile.txt'
  • parent_path: Filtered on the parent folder, e.g. 'myfolder/'

File name macro

The name of the macro which will contain the name of the current file for each file the loop iterates over, e.g. 'myfolder/myfile.txt'

File path macro

The name of the macro which will contain the full path of the current file for each file the loop iterates over, e.g. e.g. 'mybucket/myfolder/myfile.txt'

Parent path macro

The name of the macro which will contain the parent folder of the current file for each file the loop iterates over, e.g. e.g. 'myfolder/'

Recursive

If selected, the loop will also iterate over all files in all subfolders of the selected folder. Otherwise, it will only iterate over the files in the selected folder.