You are viewing the RapidMiner Server documentation for version 9.7 -Check here for latest version
Job Agents
On start, Job Agents spawn a configurable amount of Job Containers as separate system processes. Job Agents are responsible for redirecting incoming jobs from their assigned queue to the locally spawned Job Containers via REST communication. The spawned Job Containers serve solely as executing unit for RapidMiner processes. Job Containers are kept alive as long as the Job Agent application runs and will automatically shut down when the managing Job Agent shuts down.
This page outlines how to configure a Job Agent. Please refer to thearchitecturepage to read more about the Job Agent and Job Container structure overview.
Configuration
Agent Properties
You can alter the configuration of the Job Agent by changing{homeDir}/config/agent.properties．每个属性有一个评论,解释什么effect the configuration has on the agent and the spawned Job Containers. Besides settings like the amount of spawned Job Containers or the maximum amount of used memory per Job Container, it is also possible to configure more complex behavior outlined in the following sub-sections.
Container ports
When Job Containers are spawned as separate system processes during the Job Agent's startup, they are bound to system ports. This is necessary because the Job Agent communicates via REST endpoint with them, e.g. to redirect jobs or to retrieve a job's latest status. The Job Agent will use successive ports beginning from the defined starting port. The last port is determined by the amount of configured Job Containers, e.g. if1000is defined as starting port and four Job Containers should be spawned in total, the Job Agent will bind the ports1000,1001,1002and1003．Job Containers only listen locally and are unreachable from anywhere else thanlocalhost/127.0.0.1．
Container restart policies
By default, Job Containers will run indefinitely and not restart after a job has been executed. With this behavior, it's possible to execute huge amounts of jobs nearly instantly. A possible downside is that jobs might have an effect on each other when run sequentially. To overcome this, it is possible to assign restart policies to Job Containers. Supported restart policies are: run indefinitely, terminate after a configurable amount of executed processes and restart on a regular basis via cron expression. When a restart is invoked, the currently active job execution will be finished before the job containers are restarted. To change this behavior, it's possible to set thejobagent.restart.timeoutproperty. The Job Agent will then kill the Job Containers forcibly after the execution time exceeds this timeout regardless if it's still running.
Container caching for Projects
When a process from aProjectis executed by a Job Container, the Job Container will first download the corresponding project files in order to use them during execution. After the process finishes, those temporary working files are deleted. If process execution changes project files, they are automatically added as a new Snapshot.
Because Job Containers need to download project files, it will take time for large projects and thus process execution time might increase due to this. In order to reduce this initial time to download files, each Job Container caches already downloaded Projects by applying a caching strategy. In theagent.propertiesfile, this behavior can be adjusted by changing the value of thejobagent.container.repository.caching.strategyproperty. By default, a Job Container will keep two Projects in cache and replace the least recently used one when a new Project needs to be downloaded. For more details about which different strategies exist and how they can be configured to fit your needs, please have a look at the descriptions of the properties provided in theagent.propertiesfile.
Graceful Job Agent shutdown
Job Agents will by default wait for all job executions to finish before shutting down. This can however be avoided by setting the Job Agent'sjobagent.shutdown.timeout．
Container Properties
The content inside{homeDir}/config/rapidminer/is going to be copied into each job directory before container start:
{homeDir}/config/rapidminer/.RapidMiner: Defines theexecution context．Place your own configuration files into this directory if you need it for your extensions.{homeDir}/config/rapidminer/rapidminer.properties: the properties in this file are going to be set as system properties for the Job Container. The agent will also add RapidMiner Server's address properties to this file automatically before the container starts.Use this file to provide your custom properties (e.g. extension properties) for the container.
You can also use theCentral resource managementto synchronize theexecution contextfrom theRapidMiner Server home folder．
乐鱼体育安装
To enable correct execution of RapidMiner processes the Job Agent uses various external resources like JDBC drivers, RapidMiner extensions, custom Java libraries, and RapidMiner Server licenses. These resources are stored within the{homeDir}/resources/folder of the Job Agent.
Central resource management
The Job Agent's external resources are centrally managed and automatically synchronized from the RapidMiner Server instance the Job Agent is connected to. Also theExecution contextwill be synchronized by default to all Job Agents.
Resource management
Centrally managed Job Agent resources are stored in theresource/folder of theRapidMiner Server home folder．Both RapidMiner Server and all connected Job Agents use the same set of resources.
To install a new or manage an existing centrally managed resources do the following:
- 选择您想要的资源类型date (JDBC, Extensions, Custom libraries)
- From the table below locate the path of the resource type you want to update
- Update the resource type by adding or removing content from the selected folder
- Restart RapidMiner Server. All connected Job Agents will automatically synchronize the new resource configuration from RapidMiner Server.
| Type | Path |
|---|---|
| JDBC | |
| Extensions | |
| Custom libraries | |
| Licenses | Licenses are automatically synced to connected Job Agents on license installation via the RapidMiner Server UI |
| Execution context | |
Automatic synchronization
By default, all resources and the execution context are automatically synchronized from the RapidMiner Server instance after the Job Agent has been started. Also, the resources are synchronized after a restart of the RapidMiner Server instance.
The Job Agent downloads the resources to the type specific resource folder and overwrites all potential existing files. These type specific resource folders are:
| Type | Path |
|---|---|
| JDBC | |
| Extensions | |
| Custom libraries | |
| Licenses | |
| Execution context | |
Individual resource management
To setup a Job Agent with an individual resources that differ from the centrally managed resource set the automatic synchronization can be disabled by settingjobagent.sync.enabled = falsein theconfig/agent.propertiesfile. After a restart of the Job Agent it will only use the resources that are already available in the respective resource folders.
To install a new or manage an existing local resources do the following:
- Shutdown the Job Agent
- Locate the path of the resource type you want to update
- Update the resource type by adding or removing content from the selected folder
- Restart the Job Agent
Resource types
JDBC
JDBC connections can be defined in the{homeDir}/resources/jdbc/jdbc_properties.xmlfile.
Extensions
Extensions are provided from{homeDir}/resources/extensions/目录中。
Custom libraries
Custom libraries are Java libraries which you can be used in a RapidMiner process, for example within theExecute Scriptoperator. You can add these libraries to the{homeDir}/resources/libs/folder and then they are automatically available for execution.
Don't confuse custom libraries withJDBC driversorextensions．
Licenses
The Job Agent licenses are installed in the{homeDir}/resources/licenses/目录中。
You can define the number of spawned Job Containers (jobagent.container.count) for each Job Agent and the memory per Job Container (jobagent.container.memoryLimit) in the{homeDir}/config/agent.propertiesfile. Keep in mind that these settings need to comply with your current server license.
Execution context
TheExecution contextfor each job are placed in{homeDir}/config/rapidminer/.RapidMiner．
