Rapidminer to HPC

icsdm16069icsdm16069 MemberPosts:3Contributor I
inHelp
Hello,

i would like to ask if i could run the rapidminer (using linux distribution) in an HPC with slurm. I am really confused because i am trying to run it and have memory problems. My rapidminer process is working well and is inside a loop. Is anyone that have the same problem?

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University ProfessorPosts:3,389RM Data Scientist
    how much ram did you give it?
    BR,
    Martin
    - Head of Data Science Services at RapidMiner -
    Dortmund, Germany
  • icsdm16069icsdm16069 MemberPosts:3Contributor I
    my slurm program is the following:

    #！/bin/bash - l  #################################### # slurm script template # # # # Submit script: sbatch filename # # # #################################### #SBATCH --job-name=rapidminer # Job name #SBATCH --output=rapidminer.%j.out # Stdout (%j expands to jobId) #SBATCH --error=rapidminer.%j.err # Stderr (%j expands to jobId) #SBATCH --ntasks=3 # Number of tasks(processes) #SBATCH --nodes=1 # Number of nodes requested #SBATCH --ntasks-per-node=3 # Tasks per node #SBATCH --cpus-per-task=1 # Threads per task #SBATCH --time=10:10:30 # walltime #SBATCH --mem=64G # memory per NODE #SBATCH --partition=task # Partition #SBATCH --account=pa # Replace with your system project if [ x$SLURM_CPUS_PER_TASK == x ]; then export OMP_NUM_THREADS=1 else export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK fi ## LOAD MODULES ## module purge # clean up loaded modules # load necessary modules module load gnu/4.9.2 module load intel/15.0.3 module load intelmpi/5.0.3 module load cuda/8.0.61 module load java/1.8.0 module load python/3.5.0 ## RUN YOUR PROGRAM ## srun ./rapidminer-studio/scripts/rapidminer-batch.sh -f /rapidminercodes/xml.rmp

  • icsdm16069icsdm16069 MemberPosts:3Contributor I
    ok i fixed it. The first thing that you need to modify is the max memory that can be used from rapidminer and the second is the making an mpi program in order to split your input data and run the rapidminer in parallel.
    MartinLiebig
Sign InorRegisterto comment.