Hugonweb | CMSSW Parallel Generation

To do parallel generation in CMSSW, each parallel process must have different run numbers and random seeds. The following page shows how to set them by an enviromental variable supplied by the torque batch system.

Torque Multiple Jobs

Add this to the torque submission file for multiple jobs:

#Multiple Job Submission:
#Jobs will have a variable called $PBS_ARRAYID
#that will be one of the following numbers
#PBS -t 1-10

cmsDriver.py for Generation

#!/bin/bash

CONFIG_FRAGMENT="PYTHIA6_SM_ZH_HToBB_mH125_8TeV_cff.py"
BASENAME="ZH_HToBB_M125_8TeV-pythia6"

CONFIG_DIR=Configuration/GenProduction/python/
FILENAME_RAW=$BASENAME"-GEN_SIM_DIGI_HLT_RAW.root"
FILENAME_RECO=$BASENAME"-RECO.root"
PYTHON_NAME_RAW=$BASENAME"_GEN_SIM_DIGI_HLT_RAW_cfg.py"
PYTHON_NAME_RECO=$BASENAME"_RAW2DIGI_RECO_cfg.py"
N_EVENTS=1000
PILEUP=NoPileUp

echo "Config Fragment:"
echo $CONFIG_FRAGMENT

echo `date`
echo "Creating Python Config Files!"

#For Full Simulation
cmsDriver.py $CONFIG_DIR$CONFIG_FRAGMENT -s GEN,SIM,DIGI,L1,DIGI2RAW,HLT --pileup=$PILEUP --geometry DB --conditions=auto:startup --eventcontent=RAWSIM --datatier GEN-SIM-RAW -n $N_EVENTS --no_exec --fileout $FILENAME_RAW --mc --python_filename $PYTHON_NAME_RAW

#For Reconstruction 
cmsDriver.py reco -s RAW2DIGI,RECO --filein file:$FILENAME_RAW --fileout $FILENAME_RECO --conditions=auto:startup --eventcontent=RECOSIM --datatier GEN-SIM-RECO -n $N_EVENTS --no_exec --mc --python_filename $PYTHON_NAME_RECO

echo "Done."
echo "Applying my appends"

cat rawAppendText.py >> $PYTHON_NAME_RAW

cat recoAppendText.py >> $PYTHON_NAME_RECO

The script requires these python files to be in the same directory:

rawAppendText.py:

import os
import re
runNEnv =os.getenv("PBS_ARRAYID","1")
#   run number
theRun = 1000+ int(runNEnv)
#   random seed 
theSeedValue = 1234569+theRun

process.source.firstRun = cms.untracked.uint32(theRun)
process.generator.firstRun = cms.untracked.uint32(theRun)

fname = process.RAWSIMoutput.fileName
fname = re.sub(r"\.root","",fname.value())
fname += "_" + str(theRun)
fname += ".root"
process.RAWSIMoutput.fileName = fname

process.RandomNumberGeneratorService.generator.initialSeed = theSeedValue
process.RandomNumberGeneratorService.VtxSmeared.initialSeed = theSeedValue
process.RandomNumberGeneratorService.LHCTransport.initialSeed = theSeedValue
process.RandomNumberGeneratorService.hiSignalLHCTransport.initialSeed = theSeed
Value
process.RandomNumberGeneratorService.g4SimHits.initialSeed = theSeedValue
process.RandomNumberGeneratorService.mix.initialSeed = theSeedValue
process.RandomNumberGeneratorService.mixData.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simSiStripDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simSiPixelDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simEcalUnsuppressedDigis.initialSeed = the
SeedValue
process.RandomNumberGeneratorService.simHcalUnsuppressedDigis.initialSeed = the
SeedValue
process.RandomNumberGeneratorService.simMuonDTDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simMuonCSCDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simMuonRPCDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.simCastorDigis.initialSeed = theSeedValue
process.RandomNumberGeneratorService.hiSignal.initialSeed = theSeedValue
process.RandomNumberGeneratorService.hiSignalG4SimHits.initialSeed = theSeedVal
ue

recoAppendText.py:

import os
import re
runNEnv =os.getenv("PBS_ARRAYID","1")
#   run number
theRun = 1000+ int(runNEnv)

fname = process.RECOSIMoutput.fileName
fname = re.sub(r"\.root","",fname.value())
fname += "_" + str(theRun)
fname += ".root"
process.RECOSIMoutput.fileName = fname

fnames = process.source.fileNames
newfnames = []
for i in fnames:
  fname = i
  fname = re.sub(r"\.root","",fname)
  fname += "_" + str(theRun)
  fname += ".root"
  newfnames.append(fname)
process.source.fileNames = cms.untracked.vstring()
process.source.fileNames.extend(newfnames)