/usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet /usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet /usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet /usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet /usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet /usr/local/lib/python2.6/site-packages/merlin-1.7.egg/merlin/__init__.py:175: DeprecationWarning: the sets module is deprecated from sets import ImmutableSet Problem has 65 x 33 x 65 nodes per cap, 139425 nodes and 131072 elements in total memory allocation done parallel setup done node locations done velocity vars done boundary conditions done id/lm done v communications done nz radius depth rho layer 1 0.968600 0.031400 1.000000e+00 2 2 0.969600 0.030400 1.000000e+00 2 3 0.970600 0.029400 1.000000e+00 2 4 0.971600 0.028400 1.000000e+00 2 5 0.972500 0.027500 1.000000e+00 2 6 0.973500 0.026500 1.000000e+00 2 7 0.974500 0.025500 1.000000e+00 2 8 0.975500 0.024500 1.000000e+00 2 9 0.976500 0.023500 1.000000e+00 2 10 0.977400 0.022600 1.000000e+00 2 11 0.978400 0.021600 1.000000e+00 2 12 0.979400 0.020600 1.000000e+00 2 13 0.980400 0.019600 1.000000e+00 2 14 0.981400 0.018600 1.000000e+00 2 15 0.982300 0.017700 1.000000e+00 2 16 0.983300 0.016700 1.000000e+00 2 17 0.984300 0.015700 1.000000e+00 2 18 0.985300 0.014700 1.000000e+00 2 19 0.986300 0.013700 1.000000e+00 1 20 0.987200 0.012800 1.000000e+00 1 21 0.988200 0.011800 1.000000e+00 1 22 0.989200 0.010800 1.000000e+00 1 23 0.990200 0.009800 1.000000e+00 1 24 0.991200 0.008800 1.000000e+00 1 25 0.992200 0.007800 1.000000e+00 1 26 0.993100 0.006900 1.000000e+00 1 27 0.994100 0.005900 1.000000e+00 1 28 0.995100 0.004900 1.000000e+00 1 29 0.996100 0.003900 1.000000e+00 1 30 0.997100 0.002900 1.000000e+00 1 31 0.998000 0.002000 1.000000e+00 1 32 0.999000 0.001000 1.000000e+00 1 33 1.000000 0.000000 1.000000e+00 1 shape functions done mass matrix, dets done tracer setup done initial_mesh_solver_setup done initialization time = 1.919535 [ubuntu:06288] *** Process received signal *** [ubuntu:06288] Signal: Segmentation fault (11) [ubuntu:06288] Signal code: Address not mapped (1) [ubuntu:06288] Failing at address: (nil) [ubuntu:06288] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f799138fcb0] [ubuntu:06289] *** Process received signal *** [ubuntu:06288] [ 1] /lib/x86_64-linux-gnu/libc.so.6(fgets+0x1e) [0x7f7990464fee] [ubuntu:06288] [ 2] /usr/local/bin/mpipycitcoms() [0x45bf98] [ubuntu:06288] [ 3] /usr/local/bin/mpipycitcoms(initialize_tracers+0x645) [0x45cac5] [ubuntu:06288] [ 4] /usr/local/bin/mpipycitcoms(pyCitcom_ic_init_tracer_composition+0x5d) [0x420dad] [ubuntu:06288] [ 5] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x4e65) [0x4fdd55] [ubuntu:06288] [ 6] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06288] [ 7] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06288] [ 8] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06288] [ 9] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06288] [10] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06288] [11] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06288] [12] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06288] [13] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06288] [14] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06288] [15] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06288] [16] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06288] [17] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06288] [18] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06288] [19] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06288] [20] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06288] [21] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06288] [22] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06288] [23] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06288] [24] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06288] [25] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] Signal: Segmentation fault (11) [ubuntu:06289] Signal code: Address not mapped (1) [ubuntu:06289] Failing at address: (nil) [ubuntu:06288] [26] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06288] [27] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06288] [28] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06288] [29] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f00b724ecb0] [ubuntu:06290] *** Process received signal *** [ubuntu:06290] Signal: Segmentation fault (11) [ubuntu:06290] Signal code: Address not mapped (1) [ubuntu:06290] Failing at address: (nil) [ubuntu:06290] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f6e95d29cb0] [ubuntu:06290] [ 1] /lib/x86_64-linux-gnu/libc.so.6(fgets+0x1e) [0x7f6e94dfefee] [ubuntu:06290] [ 2] /usr/local/bin/mpipycitcoms() [0x45bf98] [ubuntu:06290] [ 3] /usr/local/bin/mpipycitcoms(initialize_tracers+0x645) [0x45cac5] [ubuntu:06290] [ 4] /usr/local/bin/mpipycitcoms(pyCitcom_ic_init_tracer_composition+0x5d) [0x420dad] [ubuntu:06290] [ 5] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x4e65) [0x4fdd55] [ubuntu:06290] [ 6] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06290] [ 7] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06290] [ 8] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06290] [ 9] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06290] [10] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06290] [11] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06290] [12] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06290] [13] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06290] [14] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06290] [15] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06290] [16] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06290] [17] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06290] [18] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06290] [19] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06290] [20] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06290] [21] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06290] [22] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06290] [23] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06290] [24] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06290] [25] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06290] [26] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06290] [27] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06290] [28] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06290] [29] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06290] *** End of error message *** [ubuntu:06291] *** Process received signal *** [ubuntu:06291] Signal: Segmentation fault (11) [ubuntu:06291] Signal code: Address not mapped (1) [ubuntu:06291] Failing at address: (nil) [ubuntu:06288] *** End of error message *** [ubuntu:06289] [ 1] /lib/x86_64-linux-gnu/libc.so.6(fgets+0x1e) [0x7f00b6323fee] [ubuntu:06289] [ 2] /usr/local/bin/mpipycitcoms() [0x45bf98] [ubuntu:06289] [ 3] /usr/local/bin/mpipycitcoms(initialize_tracers+0x645) [0x45cac5] [ubuntu:06289] [ 4] /usr/local/bin/mpipycitcoms(pyCitcom_ic_init_tracer_composition+0x5d) [0x420dad] [ubuntu:06289] [ 5] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x4e65) [0x4fdd55] [ubuntu:06289] [ 6] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06289] [ 7] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06289] [ 8] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06289] [ 9] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06289] [10] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06289] [11] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06289] [12] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06289] [13] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] [14] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06289] [15] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06289] [16] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06289] [17] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] [ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f1498268cb0] [ubuntu:06291] [ 1] /lib/x86_64-linux-gnu/libc.so.6(fgets+0x1e) [0x7f149733dfee] [ubuntu:06291] [ 2] /usr/local/bin/mpipycitcoms() [0x45bf98] [ubuntu:06291] [ 3] /usr/local/bin/mpipycitcoms(initialize_tracers+0x645) [0x45cac5] [ubuntu:06291] [ 4] /usr/local/bin/mpipycitcoms(pyCitcom_ic_init_tracer_composition+0x5d) [0x420dad] [ubuntu:06291] [ 5] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x4e65) [0x4fdd55] [ubuntu:06291] [ 6] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06291] [ 7] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06291] [ 8] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06291] [ 9] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06291] [10] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x5b18) [0x4fea08] [ubuntu:06291] [11] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06291] [12] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06291] [13] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] [14] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06291] [15] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06291] [16] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06291] [17] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] [18] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06291] [19] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06291] [20] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06291] [21] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] [22] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06291] [23] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06291] [24] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06291] [25] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] [26] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06291] [27] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06291] [28] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06291] [29] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06291] *** End of error message *** [ubuntu:06289] [18] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06289] [19] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06289] [20] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06289] [21] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] [22] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06289] [23] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06289] [24] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06289] [25] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] [26] /usr/local/bin/mpipycitcoms(PyEval_EvalFrameEx+0x3116) [0x4fc006] [ubuntu:06289] [27] /usr/local/bin/mpipycitcoms(PyEval_EvalCodeEx+0x80f) [0x4ff61f] [ubuntu:06289] [28] /usr/local/bin/mpipycitcoms() [0x55921b] [ubuntu:06289] [29] /usr/local/bin/mpipycitcoms(PyObject_Call+0x47) [0x484177] [ubuntu:06289] *** End of error message *** -------------------------------------------------------------------------- mpirun noticed that process rank 1 with PID 6289 on node ubuntu exited on signal 11 (Segmentation fault). --------------------------------------------------------------------------