Tune Memory
当管理更大的德ployments with multiple concurrent users and/or large workloads, resource allocation needs to be considered to avoid disruptions in daily operations.
This page lists how you can fine-tune the resources used by various components in your RapidMiner platform deployment, and what are the typical scenarios when you need to do so.
To avoid resource starvation, please double check that the total amount of memory configured/needed for the Platform does not exceed the physical memory limits of the host machine.
Tuning Single Machine Deployments
Adjusting memory settings for Job Agents
By default, RapidMiner Job Agents are configured to spawn 2 job containers, each using a maximum of 2 GBs of memory.
If you wish to increase the number of parallel RapidMiner process executions, you should either scale up the number of Job Agents, or configure a higher number of Job Containers in your Job Agent.
If you are running processes which need a large amount of memory, you need to increase the amount of memory allocated for a Job Container.
The number of Job Containers is controlled by theJOBAGENT_CONTAINER_COUNTenvironment variable of the Job Agent container. The amount of memory a Job Container can use is controlled by theJOBAGENT_CONTAINER_MEMORYLIMITenvironment variable of the Job Agent container.
To change these, you can either use Docker Deployment Manager, or use a terminal to connect to the host machine where the deployment is running. Then editing the.envfile, and finally issuingdocker-compose up -dto apply your changes.
This action will restart your RapidMiner Job Agent component, so make sure it doesn't affect any critical operation.
Provisioning Job Agents with varying shapes and sizes
By default, all Job Agents connect to the same,DEFAULTqueue. It's a typical use-case to have a separate queue for large jobs, and in that queue a Job Agent with large Job Container memory limits is listening for jobs to run.
To do this, you need to "clone" the definition of the Job Agent service in your platform definition file. Take therm-server-job-agent-svcdefinition and duplicate it, then change theJOBAGENT_CONTAINER_COUNT,JOB_QUEUE, andJOBAGENT_CONTAINER_MEMORYLIMITenvironment variables as needed.
You can either use Docker Deployment Manager, or use a terminal to connect to the host machine where the deployment is running. Then editing the.envfile anddocker-compose.ymlfile, and finally issuingdocker-compose up -dto apply your changes.
Afterwards you can scale this new Job Agent "flavor" as needed, separately from the other flavors.
Limiting CPU and memory for JupyterHub users
When using JupyterHub that is shipped in the RapidMiner platform, each user is configured to be limited to a single notebook container, where they can run their notebook kernels. To ensure a reasonable resource allocation, we have implemented a default maximum for used CPU cores and used memory that each notebook container can leverage.
To change the resource limit for notebook containers, you need to change therelevant Docker environment variableseither manually in yourdocker-compose.ymlor by usingDocker Deployment Manager. Make sure to restart the JupyterHub backend container after you have changed these settings.
The set memory limits will be applied toall users’notebook containers.
