ubuntu 安装slurm并配置A. 联网机器(dry-legion):准备离线包B. 服务器:建本地仓库并安装C. 主机名、munge、目录D. 读硬件并写配置E. 记账数据库F. 按顺序启动G. 账户、限额、强制记账H. 验证I. 作业脚本J. 可选项(没有它们前面也能用)常见问题I put every step we ran into one script, slurm_offline.sh, with three sub-commands. I only checked the syntax and the help output in my sandbox; I haven’t run prepare or install on a real machine. The first full run is the real test, so read the output as it goes.开始前要按机器修改的值:项目本次的值在哪里取CPUs / Sockets / Cores / Threads128 / 2 / 32 / 2slurmd -CRealMemory489000(实测 515402 的约 95%)slurmd -CGPU 数量2nvidia-smi -L集群名 / 账户名r750 / myaccount自定HOST、CLUSTER、DBPASS这三个变量要在同一个终端里使用,中途不要关掉终端。A. 联网机器(dry-legion):准备离线包mkdir-p~/slurm-debscd~/slurm-debsapt-getdownload$(apt-cachedepends--recurse--no-recommends --no-suggests\--no-conflicts --no-breaks --no-replaces --no-enhances\slurm-wlm slurm-wlm-basic-plugins munge openssl mariadb-server slurmdbd\|grep^\w|sort-u)rm-f*_i386.deb# 删掉多架构混进来的 32 位包ls*.deb|wc-l# 约 181cd~tarczf slurm-offline.tar.gz slurm-debs把slurm-offline.tar.gz拷到服务器(scp 或 U 盘)。B. 服务器:建本地仓库并安装cd~tarxzf slurm-offline.tar.gz# 仓库放在 /opt,因为 _apt 用户读不了家目录sudomkdir-p/opt/slurm-reposudocp~/slurm-debs/*.deb /opt/slurm-repo/cd/opt/slurm-reposudosh-cdpkg-scanpackages . /dev/null Packages 2/dev/null gzip -9c Packages Packages.gzsudochmod-RarX /opt/slurm-repogrep-c^Package:Packages# 约 181echodeb [trustedyes] file:/opt/slurm-repo .//tmp/local.listAPT_OPTS-o Dir::Etc::sourcelist/tmp/local.list -o Dir::Etc::sourceparts- -o APT::Get::List-Cleanup0sudoapt-get$APT_OPTSupdateapt-cache$APT_OPTSpolicy slurm-wlm# Candidate: 19.05.5-1# 不加 -y:看一眼要升级的包,有 libc6/dpkg/perl 等就先停下sudoapt-get$APT_OPTSinstallslurm-wlm slurm-wlm-basic-plugins munge openssl mariadb-server slurmdbd slurmd-V# slurm-wlm 19.05.5C. 主机名、munge、目录getent hosts$(hostname-s)||echo127.0.1.1$(hostname-s)|sudotee-a/etc/hostssudols-l/etc/munge/munge.key# 要用 sudo,普通用户读不了这个目录sudosystemctlenable--nowmunge munge-n|unmunge|grepSTATUS# Successsudomkdir-p/var/spool/slurmctld /var/spool/slurmd /var/log/slurmsudochownslurm:slurm /var/spool/slurmctld /var/spool/slurmd /var/log/slurmD. 读硬件并写配置slurmd-C|head-n1nvidia-smi-Lls/dev/nvidia*HOST$(hostname-s)# 主机名有大写也不要手改CLUSTERr750sudotee/etc/slurm-llnl/slurm.conf/dev/nullEOF ClusterName$CLUSTERSlurmctldHost$HOSTSlurmctldPort6817 SlurmdPort6818 SlurmUserslurm AuthTypeauth/munge MpiDefaultnone ReturnToService2 SlurmctldPidFile/run/slurmctld.pid SlurmdPidFile/run/slurmd.pid SlurmdSpoolDir/var/spool/slurmd StateSaveLocation/var/spool/slurmctld SlurmctldLogFile/var/log/slurm/slurmctld.log SlurmdLogFile/var/log/slurm/slurmd.log ProctrackTypeproctrack/cgroup TaskPlugintask/cgroup,task/affinity SchedulerTypesched/backfill SelectTypeselect/cons_tres SelectTypeParametersCR_Core_Memory GresTypesgpu AccountingStorageTypeaccounting_storage/slurmdbd AccountingStorageHostlocalhost AccountingStorageTRESgres/gpu JobAcctGatherTypejobacct_gather/linux JobAcctGatherFrequency30 NodeName$HOSTCPUs128 Boards1 SocketsPerBoard2 CoresPerSocket32 ThreadsPerCore2 RealMemory489000 Gresgpu:2 StateUNKNOWN PartitionNamemain Nodes$HOSTDefaultYES MaxTimeINFINITE StateUP EOFsudotee/etc/slurm-llnl/gres.conf/dev/nullEOF NodeName$HOSTNamegpu File/dev/nvidia0 NodeName$HOSTNamegpu File/dev/nvidia1 EOFsudotee/etc/slurm-llnl/cgroup.conf/dev/nullEOF CgroupAutomountyes ConstrainCoresyes ConstrainRAMSpaceyes EOFgrep-n-E^ClusterName|^SlurmctldHost|^GresTypes|^NodeName/etc/slurm-llnl/slurm.conf没有 GPU 的机器:去掉GresTypesgpu、AccountingStorageTRESgres/gpu、NodeName行里的Gresgpu:2,并且不要创建gres.conf。E. 记账数据库DBPASS$(openssl rand-hex16)sudosystemctlenable--nowmariadbsudotee/etc/mysql/mariadb.conf.d/99-slurm.cnf/dev/nullEOF [mysqld] innodb_buffer_pool_size1024M innodb_lock_wait_timeout900 innodb_log_file_size64M EOFsudosystemctl restart mariadbsudomysql-e CREATE DATABASE IF NOT EXISTS slurm_acct_db; CREATE USER IF NOT EXISTS slurmlocalhost IDENTIFIED BY $DBPASS; ALTER USER slurmlocalhost IDENTIFIED BY $DBPASS; GRANT ALL ON slurm_acct_db.* TO slurmlocalhost; FLUSH PRIVILEGES;sudotee/etc/slurm-llnl/slurmdbd.conf/dev/nullEOF AuthTypeauth/munge DbdHostlocalhost SlurmUserslurm DebugLevelinfo LogFile/var/log/slurm/slurmdbd.log PidFile/run/slurmdbd.pid StorageTypeaccounting_storage/mysql StorageHostlocalhost StorageUserslurm StoragePass$DBPASSStorageLocslurm_acct_db EOFsudochownslurm:slurm /etc/slurm-llnl/slurmdbd.confsudochmod600/etc/slurm-llnl/slurmdbd.conf# 权限不对 slurmdbd 不会启动systemctl is-active mariadb# activeF. 按顺序启动顺序不能换,先启动slurmctld会出现no remote port或Database is busy。sudosystemctl stop slurmctld slurmd2/dev/nullsudosystemctlenableslurmdbdsudosystemctl restart slurmdbdsleep5sudosacctmgr show cluster# 空表可以,报错不行sudosacctmgr-iaddcluster$CLUSTERsudosystemctlenableslurmctld slurmdsudosystemctl restart slurmctldsleep3sudosystemctl restart slurmd sinfo# idlescontrol shownode$(hostname-s)|grep-ECPUTot|RealMemory|Gres|StateG. 账户、限额、强制记账ACCTmyaccountsudosacctmgr-iaddaccount$ACCTDescriptionlocalOrganizationlocalsudosacctmgr-iadduser$USERAccount$ACCTsudosacctmgr-imodify user$USERsetDefaultAccount$ACCT# 可选限额:最多 4 个运行作业、64 个 CPU、同时 1 块 GPUsudosacctmgr-imodify user$USERsetMaxJobs4GrpTREScpu64,gres/gpu1sacctmgr show assocformatcluster,account,user,maxjobs,grptres%30# 账户建好之后再开强制,并且必须 restart(reconfigure 无效)echoAccountingStorageEnforceassociations,limits|sudotee-a/etc/slurm-llnl/slurm.confsudosystemctl restart slurmctld scontrol show config|grepAccountingStorageEnforce# associations,limitssleep10sinfo# 刚重启时可能显示 unk,等几秒其他用户:sudo sacctmgr -i add user 用户名 Account$ACCT。没关联账户的用户提交作业会被拒绝。改限额:sudosacctmgr-imodify user$USERsetGrpTREScpu64,gres/gpu2# 允许两块 GPUsudosacctmgr-imodify user$USERsetMaxJobs-1GrpTREScpu-1# 取消限制H. 验证srunhostnamesrun --cpus-per-task2nproc# 输出 2,说明 CPU 限制生效srun--mem1G --cpus-per-task1python3-cx bytearray(3*1024**3); print(allocated)# 被 oom-kill,说明内存限制生效srun--gresgpu:1 nvidia-smi-Lforiin12;dosbatch--gresgpu:1--wrapsleep 60;donesqueue-o%.6i %.4t %.30R# 一个 R,一个 PD (AssocGrpGRES)scancel-u$USERsacct--formatJobID,JobName,State,Elapsed,AllocTRES%40这台服务器上,CPU 和内存限制已验证生效。GPU 隔离没有配置,只靠CUDA_VISIBLE_DEVICES。I. 作业脚本#!/bin/bash#SBATCH --job-nametrain_on_local#SBATCH --nodes1#SBATCH --ntasks1#SBATCH --cpus-per-task4#SBATCH --mem40G # 必须写,超出会被杀#SBATCH --partitionmain#SBATCH --gresgpu:1 # 19.05 不支持 --gpus1#SBATCH --time0-00:05:00#SBATCH --output%x-%j.out#SBATCH --accountmyaccountPY/home/yin/miniconda3/envs/env/bin/python# conda env list 查路径cd/path/to/projectexportOMP_NUM_THREADS$SLURM_CPUS_PER_TASKsrun-n1$PY-utrain.py提交和查看:sbatch job.sh、squeue -u $USER、sacct -j jobid --formatJobID,State,Elapsed,MaxRSS,AllocTRES%40、scancel jobid。J. 可选项(没有它们前面也能用)漏写--mem时给默认值(不设的话没有内存限制):echoDefMemPerCPU3800|sudotee-a/etc/slurm-llnl/slurm.confsudosystemctl restart slurmctldGPU 隔离(没测试过):让作业只能看到分给它的那块卡。echoConstrainDevicesyes|sudotee-a/etc/slurm-llnl/cgroup.confsudotee/etc/slurm-llnl/cgroup_allowed_devices_file.conf/dev/nullEOF /dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia-modeset EOFsudosystemctl restart slurmd srun--gresgpu:1 nvidia-smi-L# 只应列出 1 块这个文件里不要写/dev/nvidia*。我之前的回复里写过,那样会放行所有 GPU,隔离就没有意义了。上面的版本只列出共用的控制设备。常见问题现象处理E: 无法定位软件包本地仓库没读到:确认/opt/slurm-repo里有Packages和Packages.gz,权限arXUnable to resolve ...slurm.conf 里的主机名和hostname -s不一致Gres(null)NodeName那一行少了Gresgpu:NAccountingStorageEnforce noneslurmctld没重启sinfo显示unk、down或drain等几秒;不行就sinfo -R,再sudo scontrol update nodename$(hostname -s) stateresume新作业分到一块显存已满的 GPU有 Slurm 之外的进程占着 GPU,Slurm 不知道;要求所有人都用 Slurm 提交,或暂时把那块卡从gres.conf里去掉 ubuntu 安装slurm并配置
ubuntu 安装slurm并配置A. 联网机器(dry-legion):准备离线包B. 服务器:建本地仓库并安装C. 主机名、munge、目录D. 读硬件并写配置E. 记账数据库F. 按顺序启动G. 账户、限额、强制记账H. 验证I. 作业脚本J. 可选项(没有它们前面也能用)常见问题I put every step we ran into one script, slurm_offline.sh, with three sub-commands. I only checked the syntax and the help output in my sandbox; I haven’t run prepare or install on a real machine. The first full run is the real test, so read the output as it goes.开始前要按机器修改的值:项目本次的值在哪里取CPUs / Sockets / Cores / Threads128 / 2 / 32 / 2slurmd -CRealMemory489000(实测 515402 的约 95%)slurmd -CGPU 数量2nvidia-smi -L集群名 / 账户名r750 / myaccount自定HOST、CLUSTER、DBPASS这三个变量要在同一个终端里使用,中途不要关掉终端。A. 联网机器(dry-legion):准备离线包mkdir-p~/slurm-debscd~/slurm-debsapt-getdownload$(apt-cachedepends--recurse--no-recommends --no-suggests\--no-conflicts --no-breaks --no-replaces --no-enhances\slurm-wlm slurm-wlm-basic-plugins munge openssl mariadb-server slurmdbd\|grep^\w|sort-u)rm-f*_i386.deb# 删掉多架构混进来的 32 位包ls*.deb|wc-l# 约 181cd~tarczf slurm-offline.tar.gz slurm-debs把slurm-offline.tar.gz拷到服务器(scp 或 U 盘)。B. 服务器:建本地仓库并安装cd~tarxzf slurm-offline.tar.gz# 仓库放在 /opt,因为 _apt 用户读不了家目录sudomkdir-p/opt/slurm-reposudocp~/slurm-debs/*.deb /opt/slurm-repo/cd/opt/slurm-reposudosh-cdpkg-scanpackages . /dev/null Packages 2/dev/null gzip -9c Packages Packages.gzsudochmod-RarX /opt/slurm-repogrep-c^Package:Packages# 约 181echodeb [trustedyes] file:/opt/slurm-repo .//tmp/local.listAPT_OPTS-o Dir::Etc::sourcelist/tmp/local.list -o Dir::Etc::sourceparts- -o APT::Get::List-Cleanup0sudoapt-get$APT_OPTSupdateapt-cache$APT_OPTSpolicy slurm-wlm# Candidate: 19.05.5-1# 不加 -y:看一眼要升级的包,有 libc6/dpkg/perl 等就先停下sudoapt-get$APT_OPTSinstallslurm-wlm slurm-wlm-basic-plugins munge openssl mariadb-server slurmdbd slurmd-V# slurm-wlm 19.05.5C. 主机名、munge、目录getent hosts$(hostname-s)||echo127.0.1.1$(hostname-s)|sudotee-a/etc/hostssudols-l/etc/munge/munge.key# 要用 sudo,普通用户读不了这个目录sudosystemctlenable--nowmunge munge-n|unmunge|grepSTATUS# Successsudomkdir-p/var/spool/slurmctld /var/spool/slurmd /var/log/slurmsudochownslurm:slurm /var/spool/slurmctld /var/spool/slurmd /var/log/slurmD. 读硬件并写配置slurmd-C|head-n1nvidia-smi-Lls/dev/nvidia*HOST$(hostname-s)# 主机名有大写也不要手改CLUSTERr750sudotee/etc/slurm-llnl/slurm.conf/dev/nullEOF ClusterName$CLUSTERSlurmctldHost$HOSTSlurmctldPort6817 SlurmdPort6818 SlurmUserslurm AuthTypeauth/munge MpiDefaultnone ReturnToService2 SlurmctldPidFile/run/slurmctld.pid SlurmdPidFile/run/slurmd.pid SlurmdSpoolDir/var/spool/slurmd StateSaveLocation/var/spool/slurmctld SlurmctldLogFile/var/log/slurm/slurmctld.log SlurmdLogFile/var/log/slurm/slurmd.log ProctrackTypeproctrack/cgroup TaskPlugintask/cgroup,task/affinity SchedulerTypesched/backfill SelectTypeselect/cons_tres SelectTypeParametersCR_Core_Memory GresTypesgpu AccountingStorageTypeaccounting_storage/slurmdbd AccountingStorageHostlocalhost AccountingStorageTRESgres/gpu JobAcctGatherTypejobacct_gather/linux JobAcctGatherFrequency30 NodeName$HOSTCPUs128 Boards1 SocketsPerBoard2 CoresPerSocket32 ThreadsPerCore2 RealMemory489000 Gresgpu:2 StateUNKNOWN PartitionNamemain Nodes$HOSTDefaultYES MaxTimeINFINITE StateUP EOFsudotee/etc/slurm-llnl/gres.conf/dev/nullEOF NodeName$HOSTNamegpu File/dev/nvidia0 NodeName$HOSTNamegpu File/dev/nvidia1 EOFsudotee/etc/slurm-llnl/cgroup.conf/dev/nullEOF CgroupAutomountyes ConstrainCoresyes ConstrainRAMSpaceyes EOFgrep-n-E^ClusterName|^SlurmctldHost|^GresTypes|^NodeName/etc/slurm-llnl/slurm.conf没有 GPU 的机器:去掉GresTypesgpu、AccountingStorageTRESgres/gpu、NodeName行里的Gresgpu:2,并且不要创建gres.conf。E. 记账数据库DBPASS$(openssl rand-hex16)sudosystemctlenable--nowmariadbsudotee/etc/mysql/mariadb.conf.d/99-slurm.cnf/dev/nullEOF [mysqld] innodb_buffer_pool_size1024M innodb_lock_wait_timeout900 innodb_log_file_size64M EOFsudosystemctl restart mariadbsudomysql-e CREATE DATABASE IF NOT EXISTS slurm_acct_db; CREATE USER IF NOT EXISTS slurmlocalhost IDENTIFIED BY $DBPASS; ALTER USER slurmlocalhost IDENTIFIED BY $DBPASS; GRANT ALL ON slurm_acct_db.* TO slurmlocalhost; FLUSH PRIVILEGES;sudotee/etc/slurm-llnl/slurmdbd.conf/dev/nullEOF AuthTypeauth/munge DbdHostlocalhost SlurmUserslurm DebugLevelinfo LogFile/var/log/slurm/slurmdbd.log PidFile/run/slurmdbd.pid StorageTypeaccounting_storage/mysql StorageHostlocalhost StorageUserslurm StoragePass$DBPASSStorageLocslurm_acct_db EOFsudochownslurm:slurm /etc/slurm-llnl/slurmdbd.confsudochmod600/etc/slurm-llnl/slurmdbd.conf# 权限不对 slurmdbd 不会启动systemctl is-active mariadb# activeF. 按顺序启动顺序不能换,先启动slurmctld会出现no remote port或Database is busy。sudosystemctl stop slurmctld slurmd2/dev/nullsudosystemctlenableslurmdbdsudosystemctl restart slurmdbdsleep5sudosacctmgr show cluster# 空表可以,报错不行sudosacctmgr-iaddcluster$CLUSTERsudosystemctlenableslurmctld slurmdsudosystemctl restart slurmctldsleep3sudosystemctl restart slurmd sinfo# idlescontrol shownode$(hostname-s)|grep-ECPUTot|RealMemory|Gres|StateG. 账户、限额、强制记账ACCTmyaccountsudosacctmgr-iaddaccount$ACCTDescriptionlocalOrganizationlocalsudosacctmgr-iadduser$USERAccount$ACCTsudosacctmgr-imodify user$USERsetDefaultAccount$ACCT# 可选限额:最多 4 个运行作业、64 个 CPU、同时 1 块 GPUsudosacctmgr-imodify user$USERsetMaxJobs4GrpTREScpu64,gres/gpu1sacctmgr show assocformatcluster,account,user,maxjobs,grptres%30# 账户建好之后再开强制,并且必须 restart(reconfigure 无效)echoAccountingStorageEnforceassociations,limits|sudotee-a/etc/slurm-llnl/slurm.confsudosystemctl restart slurmctld scontrol show config|grepAccountingStorageEnforce# associations,limitssleep10sinfo# 刚重启时可能显示 unk,等几秒其他用户:sudo sacctmgr -i add user 用户名 Account$ACCT。没关联账户的用户提交作业会被拒绝。改限额:sudosacctmgr-imodify user$USERsetGrpTREScpu64,gres/gpu2# 允许两块 GPUsudosacctmgr-imodify user$USERsetMaxJobs-1GrpTREScpu-1# 取消限制H. 验证srunhostnamesrun --cpus-per-task2nproc# 输出 2,说明 CPU 限制生效srun--mem1G --cpus-per-task1python3-cx bytearray(3*1024**3); print(allocated)# 被 oom-kill,说明内存限制生效srun--gresgpu:1 nvidia-smi-Lforiin12;dosbatch--gresgpu:1--wrapsleep 60;donesqueue-o%.6i %.4t %.30R# 一个 R,一个 PD (AssocGrpGRES)scancel-u$USERsacct--formatJobID,JobName,State,Elapsed,AllocTRES%40这台服务器上,CPU 和内存限制已验证生效。GPU 隔离没有配置,只靠CUDA_VISIBLE_DEVICES。I. 作业脚本#!/bin/bash#SBATCH --job-nametrain_on_local#SBATCH --nodes1#SBATCH --ntasks1#SBATCH --cpus-per-task4#SBATCH --mem40G # 必须写,超出会被杀#SBATCH --partitionmain#SBATCH --gresgpu:1 # 19.05 不支持 --gpus1#SBATCH --time0-00:05:00#SBATCH --output%x-%j.out#SBATCH --accountmyaccountPY/home/yin/miniconda3/envs/env/bin/python# conda env list 查路径cd/path/to/projectexportOMP_NUM_THREADS$SLURM_CPUS_PER_TASKsrun-n1$PY-utrain.py提交和查看:sbatch job.sh、squeue -u $USER、sacct -j jobid --formatJobID,State,Elapsed,MaxRSS,AllocTRES%40、scancel jobid。J. 可选项(没有它们前面也能用)漏写--mem时给默认值(不设的话没有内存限制):echoDefMemPerCPU3800|sudotee-a/etc/slurm-llnl/slurm.confsudosystemctl restart slurmctldGPU 隔离(没测试过):让作业只能看到分给它的那块卡。echoConstrainDevicesyes|sudotee-a/etc/slurm-llnl/cgroup.confsudotee/etc/slurm-llnl/cgroup_allowed_devices_file.conf/dev/nullEOF /dev/null /dev/urandom /dev/zero /dev/sda* /dev/cpu/*/* /dev/pts/* /dev/nvidiactl /dev/nvidia-uvm /dev/nvidia-uvm-tools /dev/nvidia-modeset EOFsudosystemctl restart slurmd srun--gresgpu:1 nvidia-smi-L# 只应列出 1 块这个文件里不要写/dev/nvidia*。我之前的回复里写过,那样会放行所有 GPU,隔离就没有意义了。上面的版本只列出共用的控制设备。常见问题现象处理E: 无法定位软件包本地仓库没读到:确认/opt/slurm-repo里有Packages和Packages.gz,权限arXUnable to resolve ...slurm.conf 里的主机名和hostname -s不一致Gres(null)NodeName那一行少了Gresgpu:NAccountingStorageEnforce noneslurmctld没重启sinfo显示unk、down或drain等几秒;不行就sinfo -R,再sudo scontrol update nodename$(hostname -s) stateresume新作业分到一块显存已满的 GPU有 Slurm 之外的进程占着 GPU,Slurm 不知道;要求所有人都用 Slurm 提交,或暂时把那块卡从gres.conf里去掉