[cig-commits] r15016 - seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona
dkomati1 at geodynamics.org
dkomati1 at geodynamics.org
Tue May 19 09:22:49 PDT 2009
Author: dkomati1
Date: 2009-05-19 09:22:49 -0700 (Tue, 19 May 2009)
New Revision: 15016
Added:
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_important_paths_on_MareNostrum.txt
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_mpitrace_explained.xml
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/mpitrace.xml
Modified:
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/Makefile_things_to_add_to_use_ParaVer_and_PAPI.txt
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/generate_trace_marenostrum.sh
seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/script_marenostrum_parallel_run_with_trace_mx.ll
Log:
updated the ParaVer tracing scripts and related README files for MareNostrum in Barcelona
Modified: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/Makefile_things_to_add_to_use_ParaVer_and_PAPI.txt
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/Makefile_things_to_add_to_use_ParaVer_and_PAPI.txt 2009-05-19 00:04:48 UTC (rev 15015)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/Makefile_things_to_add_to_use_ParaVer_and_PAPI.txt 2009-05-19 16:22:49 UTC (rev 15016)
@@ -1,14 +1,9 @@
-Here is what to add to the Makefile on MareNostrum in Barcelona
-to link with the ParaVer and PAPI tracing system (the paths below
-are OK in May 2009):
+May 2009: there is no need to add tracing libraries to the Makefile any more
+to generate some ParaVer traces. Just type on MareNostrum:
-MPITRACE_HOME = /gpfs/apps/CEPBATOOLS/mpitrace-mx/64
-PAPI_HOME = /gpfs/apps/PAPI/3.2.1-970mp/64
+./configure FC=xlf_r
-and add this at the end of the definition of FLAGS_NO_CHECK:
+and then edit the Makefile to use
+-qtune=ppc970 -qarch=ppc64v instead of -qtune=auto -qarch=auto
--L$(MPITRACE_HOME)/lib -lmpitracef -lxml2 -L${PAPI_HOME}/lib -lpapi -lperfctr /opt/osshpc/mpich-mx/64/lib/libmpich.a
-
-Also change the default compiler from "xlf" to "xlf_r".
-
Added: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_important_paths_on_MareNostrum.txt
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_important_paths_on_MareNostrum.txt (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_important_paths_on_MareNostrum.txt 2009-05-19 16:22:49 UTC (rev 15016)
@@ -0,0 +1,16 @@
+
+Right path of ParaVer, May 2009:
+/gpfs/apps/CEPBATOOLS/mpitrace-mx/64
+
+Right path of examples of ParaVer, May 2009:
+/gpfs/apps/CEPBATOOLS/mpitrace-mx/64/share/example
+
+Storage space available (do NOT use the home directory, which is limited to 20 GB):
+
+# added in May 2009: directories with a lot of free space
+## DK DK the best is to work in "projects", NOT "scratch"
+## DK DK because it is bigger and is kind of permanent
+
+alias cdprojects='pushd /gpfs/projects/bsc42/bsc42023'
+alias cdscratch='pushd /gpfs/scratch/bsc42/bsc42023'
+
Added: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_mpitrace_explained.xml
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_mpitrace_explained.xml (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/README_mpitrace_explained.xml 2009-05-19 16:22:49 UTC (rev 15016)
@@ -0,0 +1,172 @@
+<?xml version='1.0'?>
+
+<!-- Here comes the MPItrace configuration.
+ As a general rule, "enabled" means that the feature is enabled :) If
+ it's not enabled, then the value can be set to some default.
+-->
+
+<!-- Must we activate the tracing? Which is the tracing mode? (detail/bursts) Where is it located? Which kind of trace? Version of the XML parser?-->
+<trace enabled="yes"
+ home="/gpfs/apps/CEPBATOOLS/mpitrace-mx/64"
+ initial-mode="detail"
+ type="paraver"
+ xml-parser-id="Id: xml-parse.c,v 1.20 2008/05/19 13:55:11 harald Exp $"
+>
+
+ <!-- Configuration of some MPI dependant values -->
+ <mpi enabled="yes">
+ <!-- Gather counters in the MPI routines? -->
+ <counters enabled="yes" />
+ </mpi>
+
+ <!-- Emit information of the callstack -->
+ <callers enabled="yes">
+ <!-- At MPI calls, select depth level -->
+ <mpi enabled="yes">1-3</mpi>
+ <!-- At sampling points, select depth level -->
+ <sampling enabled="yes">1-5</sampling>
+ </callers>
+
+ <!-- Configuration of some OpenMP dependant values -->
+ <openmp enabled="no">
+ <!-- If the library instruments OpenMP, shall we gather info about locks?
+ Obtaining such information can make the final trace quite large.
+ -->
+ <locks enabled="no" />
+ <!-- Gather counters in the OpenMP routines? -->
+ <counters enabled="yes" />
+ </openmp>
+
+ <!-- Configuration of User Functions -->
+ <user-functions enabled="yes" list="/home/bsc41/bsc41273/user-functions.dat">
+ <!-- Set a depth limit for the UF routines -->
+ <max-depth enabled="no">3</max-depth>
+ <!-- Gather counters on the UF routines? -->
+ <counters enabled="yes" />
+ </user-functions>
+
+ <!-- Configure which software/hardware counters must be collected -->
+ <counters enabled="yes">
+ <!-- Configure the CPU hardware counters. You can define here as many sets
+ as you want. You can also define if MPI/OpenMP calls must report such
+ counters.
+ Starting-set property defines which set is chosen from every task.
+ Possible values are:
+ - cyclic : The sets are distributed in a cyclic fashion among all
+ tasks. So Task 0 takes set 1, Task 1 takes set 2,...
+ - block : The sets are distributed in block fashion among all tasks.
+ Task [0..i-1] takes set 1, Task [i..2*i-1] takes set 2, ...
+ - Number : All the tasks will start with the given set
+ (from 1..N).
+ -->
+ <cpu enabled="yes" starting-set-distribution="1">
+ <!-- In this example, we configure two sets of counters. The first will
+ be changed into the second after 5 calls to some collective
+ operation on MPI_COMM_WORLD. Once the second is activated, it will
+ turn to the first after 5seconds (aprox. depending on the MPI calls
+ granularity)
+ If you want that any set be counting forever, just don't set
+ changeat-globalops, or, changeat-time.
+
+ Each set has it's own properties.
+ domain -> in which domain must PAPI obtain the information (see
+ PAPI info)
+ changeat-globalops=num -> choose the next set after num
+ MPI_COMM_WORLD operations
+ changeat-time=numTime -> choose the next set after num Time
+ (for example 5s, 15m,..)
+ -->
+ <set enabled="yes" domain="all" changeat-globalops="5">
+ PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L1_DCM
+ <!-- Samples the application based on a frequency on a selected HWC -->
+ <sampling enabled="yes" frequency="100000000">PAPI_TOT_CYC</sampling>
+ </set>
+ <set enabled="yes" domain="user" changeat-globalops="5">
+ PAPI_TOT_INS,PAPI_FP_INS,PAPI_TOT_CYC
+ </set>
+ </cpu>
+
+ <!-- Do we want to gather information of the network counters?
+ Nowadays we can gather information about MX/GM cards.
+ -->
+ <network enabled="yes" />
+
+ <!-- Obtain resource usage information -->
+ <resource-usage enabled="yes" />
+ </counters>
+
+ <!-- Define the characteristics of the tracing storage. If not defined,
+ or set, the tracing will send the traces to the current directory
+ with a default output name.
+ -->
+ <storage enabled="no">
+ <!-- The intermediate files will take the name of the application -->
+ <trace-prefix enabled="yes">TRACE</trace-prefix>
+ <!-- Stop the tracing when the intermediate file reaches this amount of MBs -->
+ <size enabled="no">5</size>
+ <!-- Where must we store the MPIT files while the app runs?
+ Set make-dir to yes if you want to make this directory before running -->
+ <temporal-directory enabled="yes" make-dir="no">/scratch</temporal-directory>
+ <!-- Where must we store the MPIT files once the app ends?
+ Set make-dir to yes if you want to make this directory before running -->
+ <final-directory enabled="yes" make-dir="no">/gpfs/scratch/bsc41/bsc41273</final-directory>
+ <!-- Must the MPITs be gathered once the app ends into a single node? -->
+ <gather-mpits enabled="no" />
+ </storage>
+
+ <!-- Buffer configuration -->
+ <buffer enabled="yes">
+ <!-- How many events can we handle before any flush -->
+ <size enabled="yes">150000</size>
+ <!-- Use the event buffer in a circular manner? You can use this option to
+ trace the last set of events. Needs MPI global routines operating on
+ MPI_COMM_WORLD communicator to be merged
+ -->
+ <circular enabled="no" />
+ </buffer>
+
+ <!-- Control tracing -->
+ <trace-control enabled="yes">
+ <!-- We can start the application with a "latent tracing" and wake it up
+ once a control file is created. Use the property 'frequency' to
+ choose at which frequency this check must be done. If not supplied,
+ it will be checked every 100 global operations on MPI_COMM_WORLD.
+ -->
+ <file enabled="no" frequency="5m">/gpfs/scratch/bsc41/bsc41273/control</file>
+ <!--
+ -->
+ <global-ops enabled="no"></global-ops>
+ <!-- A signal can be used to terminate the tracing. Available values can be
+ only USR1/USR2 Some MPI implementations handle one of those, so check
+ first which is available to you. -->
+ <remote-control enabled="no" method="signal">USR1</remote-control>
+ </trace-control>
+
+ <!-- Other options -->
+ <others enabled="yes">
+ <!-- Want to force a minimum amount of time of tracing? Here we force 10
+ minutes -->
+ <minimum-time enabled="no">10m</minimum-time>
+ </others>
+
+ <!-- Bursts library enabled? This requires an special library! -->
+ <bursts enabled="no">
+ <!-- Specify the threshold. This is mandatory! In this example, the
+ threshold is limitted to 500 microseconds
+ -->
+ <threshold enabled="yes">500u</threshold>
+ <!-- Report MPI statistics? -->
+ <mpi-statistics enabled="yes" />
+ </bursts>
+
+ <!-- Cell BE specific tracing options -->
+ <cell enabled="no">
+ <!-- Limit the size of the intermediate files for every SPU -->
+ <spu-file-size enabled="yes">5</spu-file-size>
+ <!-- Which is the buffer size per SPU? -->
+ <spu-buffer-size enabled="yes">64</spu-buffer-size>
+ <!-- Which DMA tag to use for the tracing flushes? -->
+ <spu-dma-channel enabled="yes">2</spu-dma-channel>
+ </cell>
+
+</trace>
Modified: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/generate_trace_marenostrum.sh
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/generate_trace_marenostrum.sh 2009-05-19 00:04:48 UTC (rev 15015)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/generate_trace_marenostrum.sh 2009-05-19 16:22:49 UTC (rev 15016)
@@ -1,44 +1,10 @@
#!/bin/sh
-###########################################################################
-# Shared library preloading example (README)
-#
-# So as to trace with the dynamic library, just set LD_PRELOAD to the .so
-# MPItrace library and set the required environment variables. In this
-# example we set MPITRACE_ON to activate the tracing and the MPTRACE_COUNTERS
-# to select which counters we wanna obtain from PAPI.
-#
-###########################################################################
-
-# DK DK this below OK in May 2009
export MPITRACE_HOME=/gpfs/apps/CEPBATOOLS/mpitrace-mx/64
+export MPTRACE_CONFIG_FILE=mpitrace.xml
+
export LD_PRELOAD=${MPITRACE_HOME}/lib/libmpitrace.so
-export MPITRACE_ON=1
-# the list of available counters is in /gpfs/apps/CEPBATOOLS/mpitrace/64/share/example-with-user-events/counters.txt
-# monitor L1 below
-#export MPTRACE_COUNTERS="0x80000000,0x80000017,0x80000032,0x8000003b"
-# monitor L2 below
-export MPTRACE_COUNTERS="0x80000002,0x80000019,0x80000032,0x8000003b"
-# 0x80000000 is PAPI_L1_DCM # cannot do both L1 and L2 at the same time, need to start two runs
-# 0x80000017 is PAPI_L1_LDM
-# 0x80000002 is PAPI_L2_DCM
-# 0x80000019 is PAPI_L2_LDM
-# 0x80000032 is PAPI_TOT_INS
-# 0x8000003b is PAPI_TOT_CYC
-#
-export MPITRACE_MPI_COUNTERS_ON=1
-export MPTRACE_COUNTERS_DOMAIN=all
-export MPITRACE_MPI_CALLER=1,2,3
-#########export MPTRACE_CONFIG_FILE=mpitrace_extended.xml
-export MPTRACE_BUFFER_SIZE=150000
-# older from the year 2006: begin
-#export MPTRACE_DIR=/gpfs/scratch/hpce07/hpce07084/TRACE_DIR/
-#export MPTRACE_FILE_SIZE=10
-#export MPTRACE_BUFFER_SIZE=25000
-#export MPTRACE_COUNTERS="0x80000004,0x8000000e,0x80000032,0x8000003b"
-# older from the year 2006: end
-
## Run the desired program
$*
Added: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/mpitrace.xml
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/mpitrace.xml (rev 0)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/mpitrace.xml 2009-05-19 16:22:49 UTC (rev 15016)
@@ -0,0 +1,78 @@
+<?xml version='1.0'?>
+
+<trace enabled="yes"
+ home="/gpfs/apps/CEPBATOOLS/mpitrace-mx/64"
+ initial-mode="detail"
+ type="paraver"
+ xml-parser-id="Id: xml-parse.c,v 1.20 2008/05/19 13:55:11 harald Exp $"
+>
+ <mpi enabled="yes">
+ <counters enabled="yes" />
+ </mpi>
+
+ <openmp enabled="no">
+ <locks enabled="no" />
+ <counters enabled="yes" />
+ </openmp>
+
+ <callers enabled="yes">
+ <mpi enabled="yes">1-3</mpi>
+ <sampling enabled="no">1-5</sampling>
+ </callers>
+
+ <user-functions enabled="no" list="/home/bsc41/bsc41273/user-functions.dat">
+ <max-depth enabled="no">3</max-depth>
+ <counters enabled="yes" />
+ </user-functions>
+
+ <counters enabled="yes">
+ <cpu enabled="yes" starting-set-distribution="1">
+ <set enabled="yes" domain="all" changeat-globalops="0">
+ PAPI_TOT_INS,PAPI_TOT_CYC,PAPI_L1_DCM
+ <sampling enabled="no" frequency="100000000">PAPI_TOT_CYC</sampling>
+ </set>
+ <set enabled="yes" domain="user" changeat-globalops="0">
+ PAPI_TOT_INS,PAPI_FP_INS,PAPI_TOT_CYC
+ </set>
+ </cpu>
+
+ <network enabled="yes" />
+
+ <resource-usage enabled="yes" />
+ </counters>
+
+ <storage enabled="no">
+ <trace-prefix enabled="yes">TRACE</trace-prefix>
+ <size enabled="no">5</size>
+ <temporal-directory enabled="yes" make-dir="no">/scratch</temporal-directory>
+ <final-directory enabled="yes" make-dir="no">/gpfs/scratch/bsc41/bsc41273</final-directory>
+ <gather-mpits enabled="no" />
+ </storage>
+
+ <buffer enabled="yes">
+ <size enabled="yes">250000</size>
+ <circular enabled="no" />
+ </buffer>
+
+ <trace-control enabled="no">
+ <file enabled="no" frequency="5m">/gpfs/scratch/bsc41/bsc41273/control</file>
+ <global-ops enabled="no"></global-ops>
+ <remote-control enabled="no" method="signal">USR1</remote-control>
+ </trace-control>
+
+ <others enabled="no">
+ <minimum-time enabled="no">10m</minimum-time>
+ </others>
+
+ <bursts enabled="no">
+ <threshold enabled="yes">500u</threshold>
+ <mpi-statistics enabled="yes" />
+ </bursts>
+
+ <cell enabled="no">
+ <spu-file-size enabled="yes">5</spu-file-size>
+ <spu-buffer-size enabled="yes">64</spu-buffer-size>
+ <spu-dma-channel enabled="yes">2</spu-dma-channel>
+ </cell>
+
+</trace>
Modified: seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/script_marenostrum_parallel_run_with_trace_mx.ll
===================================================================
--- seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/script_marenostrum_parallel_run_with_trace_mx.ll 2009-05-19 00:04:48 UTC (rev 15015)
+++ seismo/3D/SPECFEM3D_GLOBE/trunk/UTILS/MareNostrum_Barcelona/script_marenostrum_parallel_run_with_trace_mx.ll 2009-05-19 16:22:49 UTC (rev 15016)
@@ -16,23 +16,19 @@
#####################################################################
#@ total_tasks = 32
#@ tasks_per_node = 4
-# Wall clock limit hhh:mm:ss
-######### hpce is a queue that has a 24-hour (i.e. 86400 seconds) wallclock time limit
+## Wall clock limit hhh:mm:ss
#@ wall_clock_limit = 00:20:00
#@ output = Specfem3D_run_%j.out
#@ error = Specfem3D_run_%j.err
#@ queue
#@ features = mx
-# DK DK this below OK in May 2009
-MPITRACE_HOME=/gpfs/apps/CEPBATOOLS/mpitrace-mx/64
-
#environment
MP_EUILIB=mx
OBJECT_MODE=64
MP_RSH=ssh
- srun ./generate_trace_paraver.sh ./xspecfem2D
+ srun ./generate_trace_paraver.sh ./xspecfem3D
# then merge the trace at the end
sleep 5
More information about the CIG-COMMITS
mailing list