[CIG-SHORT] Fwd: PyLith-build test fails
Matthew Knepley
knepley at mcs.anl.gov
Tue Mar 24 15:30:13 PDT 2015
Moving this discussion to petsc-maint, since you seem to have a completely
broken MPI install.
Can you send us $PETSC_DIR/configure.log and make.log.
Background: The run was segfaulting on the IBarrier setup so I had him
remove that #define and rebuild. Now it is
faulting on the normal Allreduce. Has anyone seen this before?
Thanks,
Matt
---------- Forwarded message ----------
From: Mamun Miah <mmiah at go.olemiss.edu>
Date: Tue, Mar 24, 2015 at 5:17 PM
Subject: Re: [CIG-SHORT] PyLith-build test fails
To: cig-short at geodynamics.org
Hi Matt,
After doing 'make clean' first and 'make' next in the PETSC_DIR I was able
to get rid of the IBARRIER and rebuild PETSc. Then I get the following
errors after I run a sample problem. I checked this website as suggested in
the message below:
http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind.
<http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind>
I have tried reconfiguring and building with '--download-mpich' in the
PETSC_DIR to see if it changes anything but no change at all. I have also
checked the FAQ (2nd link in the error messages) page for memory corruption
but could not figure out much from there.
[0]PETSC ERROR: PetscTrFreeDefault() called from
PetscCommBuildTwoSided_Allreduce() line 179 in
/home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
[0]PETSC ERROR: Block [id=0(64)] at address 0x301e920 is corrupted
(probably write past end of array)
[0]PETSC ERROR: Block allocated in PetscCommBuildTwoSided_Allreduce() line
168 in /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Memory corruption:
http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind
[0]PETSC ERROR: Corrupted memory
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2251-gb193602 GIT
Date: 2015-02-19 19:20:25 -0600
[0]PETSC ERROR: /home/lawrence/pylith/bin/mpinemesis on a arch-pylith named
petaluma by lawrence Tue Mar 24 11:03:51 2015
[0]PETSC ERROR: Configure options --prefix=/home/lawrence/pylith
--with-c2html=0 --with-x=0 --with-clanguage=C --with-mpicompilers=1
--with-debugging=0 --with-shared-libraries=1 --download-chaco=1
--download-ml=1 --download-fblaslapack=1 --with-hdf5=1
--with-hdf5-include=/home/lawrence/pylith/include
--with-hdf5-lib=/home/lawrence/pylith/lib/libhdf5.dylib --LIBS=-lz
CPPFLAGS="-I/home/lawrence/pylith/include "
LDFLAGS="-L/home/lawrence/pylith/lib " CFLAGS="-g -O2" CXXFLAGS="-g -O2
-DMPICH_IGNORE_CXX_SEEK" FCFLAGS="-g -O2"
PETSC_DIR=/home/lawrence/build/pylith/petsc-pylith PETSC_ARCH=arch-pylith
[0]PETSC ERROR: #1 PetscTrFreeDefault() line 289 in
/home/lawrence/build/pylith/petsc-pylith/src/sys/memory/mtr.c
[0]PETSC ERROR: #2 PetscCommBuildTwoSided_Allreduce() line 179 in
/home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
[0]PETSC ERROR: #3 PetscCommBuildTwoSided() line 237 in
/home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
[0]PETSC ERROR: #4 PetscSFSetUp_Basic() line 552 in
/home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/impls/basic/sfbasic.c
[0]PETSC ERROR: #5 PetscSFSetUp() line 192 in
/home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
[0]PETSC ERROR: #6 PetscSFReduceBegin() line 990 in
/home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
[0]PETSC ERROR: #7 DMLocalToGlobalBegin() line 1893 in
/home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
[0]PETSC ERROR: #8 void pylith::topology::Field::complete()() line 710 in
../../../pylith-2.1.0/libsrc/pylith/topology/Field.cc
On Fri, Mar 20, 2015 at 11:09 AM, Matthew Knepley <knepley at mcs.anl.gov>
wrote:
> On Fri, Mar 20, 2015 at 11:34 AM, Mamun Miah <mmiah at go.olemiss.edu> wrote:
>
>> Dear Matt,
>>
>> I have run a sample problem from PyLith tutorial and got the following
>> PETSc errors:
>>
>> [0]PETSC ERROR: #1 PetscCommBuildTwoSided_Ibarrier() line 122 in
>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>> [0]PETSC ERROR: #2 PetscCommBuildTwoSided() line 231 in
>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>> [0]PETSC ERROR: #3 PetscSFSetUp_Basic() line 552 in
>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/impls/basic/sfbasic.c
>> [0]PETSC ERROR: #4 PetscSFSetUp() line 192 in
>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>> [0]PETSC ERROR: #5 PetscSFReduceBegin() line 990 in
>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>> [0]PETSC ERROR: #6 DMLocalToGlobalBegin() line 1893 in
>> /home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
>> [0]PETSC ERROR: #7 void pylith::topology::Field::complete()() line 710 in
>> ../../../pylith-2.1.0/libsrc/pylith/topology/Field.cc
>> Fatal error. Calling MPI_Abort() to abort PyLith application.
>>
>> Looks like the IBARRIER is still causing the problem.
>>
>
> This makes it clear that you did not rebuild PETSc. Here is the source:
>
>
> https://bitbucket.org/petsc/petsc/src/6d5951fb15cef55c0a1c902bab2c56b27935b002/src/sys/utils/mpits.c?at=master#cl-86
>
> The function listed will not even be compiled if you rebuilt with that
> #ifdef removed. Please
> remove that statement from petscconf.h, rebuild PETSc, which will recreate
> the shared library.
>
> Thanks,
>
> Matt
>
>
>> On Thu, Mar 19, 2015 at 3:30 PM, Matthew Knepley <knepley at mcs.anl.gov>
>> wrote:
>>
>>> On Thu, Mar 19, 2015 at 4:07 PM, Mamun Miah <mmiah at go.olemiss.edu>
>>> wrote:
>>>
>>>> Deat Matt,
>>>>
>>>> I have tried rebuilding PETSc after removing HAVE_MPI_IBARRIER from the
>>>> petscconf.h file. It did not show any error. Then I went on building in the
>>>> pylith directory. It shows the message as follows-
>>>>
>>>
>>> This is fine. PyLith does not need to be rebuilt for this change.
>>>
>>>
>>>> lawrence at petaluma:~/build/pylith$ make
>>>> Making all in bin
>>>> make[1]: Entering directory `/home/lawrence/build/pylith/bin'
>>>> make[1]: Nothing to be done for `all'.
>>>> make[1]: Leaving directory `/home/lawrence/build/pylith/bin'
>>>> make[1]: Entering directory `/home/lawrence/build/pylith'
>>>> make[1]: Nothing to be done for `all-am'.
>>>> make[1]: Leaving directory `/home/lawrence/build/pylith'
>>>> lawrence at petaluma:~/build/pylith$
>>>>
>>>> Also, rebuilding PETSc after removing IBARRIER does not seem to have
>>>> any effect on the $PETSC_DIR/config.log file.
>>>>
>>>
>>> Also fine, you did not reconfigure.
>>>
>>> Please run your example again and see if it works.
>>>
>>> Thanks,
>>>
>>> Matt
>>>
>>>
>>>> Thanks,
>>>> Mamun
>>>>
>>>> On Wed, Mar 18, 2015 at 1:57 PM, Matthew Knepley <knepley at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>> On Wed, Mar 18, 2015 at 1:24 PM, Brad Aagaard <baagaard at usgs.gov>
>>>>> wrote:
>>>>>
>>>>>> The test-suite.log file lists the following error:
>>>>>>
>>>>>
>>>>> Yes, it is possible that we are finding the MPI_Ibarrier() function in
>>>>> our test, but the implementation
>>>>> actually just crashes. We have seen this before with incomplete MPI
>>>>> implementations. You can
>>>>> remove
>>>>>
>>>>> HAVE_MPI_IBARRIER
>>>>>
>>>>> from
>>>>>
>>>>> $PETSC_DIR/$PETSC_ARCH/include/petscconf.h
>>>>>
>>>>> and rebuild PETSc using
>>>>>
>>>>> cd $PETSC_DIR
>>>>> make
>>>>>
>>>>> Please send
>>>>>
>>>>> $PETSC_DIR/configure.log
>>>>>
>>>>> so I can look at the test for that function.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt
>>>>>
>>>>>
>>>>>> test_writeData (TestOutputSolnPoints.TestOutputSolnPoints) ...
>>>>>> [0]PETSC ERROR: #1 PetscCommBuildTwoSided_Ibarrier() line 122 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>>>>> [0]PETSC ERROR: #2 PetscCommBuildTwoSided() line 231 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>>>>> [0]PETSC ERROR: #3 PetscSFSetUp_Basic() line 552 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/
>>>>>> impls/basic/sfbasic.c
>>>>>> [0]PETSC ERROR: #4 PetscSFSetUp() line 192 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>>>>>> [0]PETSC ERROR: #5 PetscSFBcastBegin() line 923 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>>>>>> [0]PETSC ERROR: #6 DMGlobalToLocalBegin() line 1692 in
>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
>>>>>> [0]PETSC ERROR: #7 void pylith::topology::Field::scatterGlobalToLocal(PetscVec,
>>>>>> const char*) const() line 1168 in ../../../pylith-2.1.0/libsrc/
>>>>>> pylith/topology/Field.cc
>>>>>> ERROR
>>>>>>
>>>>>> ============================================================
>>>>>> ==========
>>>>>> ERROR: test_writeData (TestOutputSolnPoints.TestOutputSolnPoints)
>>>>>> ------------------------------------------------------------
>>>>>> ----------
>>>>>> Traceback (most recent call last):
>>>>>> File "/home/lawrence/build/pylith/pylith-2.1.0/unittests/
>>>>>> pytests/meshio/TestOutputSolnPoints.py", line 194, in test_writeData
>>>>>> output.writeData(2.0, self.fields)
>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/
>>>>>> pylith/meshio/OutputManager.py", line 252, in writeData
>>>>>> self._appendVertexField(t, field, mesh)
>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/
>>>>>> pylith/meshio/OutputManager.py", line 439, in _appendVertexField
>>>>>> ModuleOutputManager.appendVertexField(self, t, field, mesh)
>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/pylith/meshio/meshio.py",
>>>>>> line 300, in appendVertexField
>>>>>> def appendVertexField(self, *args): return _meshio.OutputManager_appendVertexField(self,
>>>>>> *args)
>>>>>> RuntimeError: Error detected while in PETSc function.
>>>>>>
>>>>>> This looks like it may be an MPI issue. I don't get this error on any
>>>>>> of the machines I have tested using OpenMPI and MPICH. What version of MPI
>>>>>> are you using? How was MPI installed (installer, sys admin, you, someone
>>>>>> else)?
>>>>>>
>>>>>> Regards,
>>>>>> Brad
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 03/18/2015 11:52 AM, Mamun Miah wrote:
>>>>>>
>>>>>>> Dear developers,
>>>>>>>
>>>>>>> I have been trying to build PyLith from the source code but
>>>>>>> apparently
>>>>>>> reached an impasse. I am getting the following test failure.
>>>>>>>
>>>>>>> FAIL: testmeshio.py
>>>>>>>
>>>>>>> ============================================================
>>>>>>> ================
>>>>>>> Testsuite summary for PyLith 2.1.0
>>>>>>> ============================================================
>>>>>>> ================
>>>>>>> # TOTAL: 3
>>>>>>> # PASS: 2
>>>>>>> # SKIP: 0
>>>>>>> # XFAIL: 0
>>>>>>> # FAIL: 1
>>>>>>> # XPASS: 0
>>>>>>> # ERROR: 0
>>>>>>> ============================================================
>>>>>>> ================
>>>>>>> See unittests/pytests/meshio/test-suite.log
>>>>>>> Please report to cig-short at geodynamics.org
>>>>>>> ============================================================
>>>>>>> ================
>>>>>>> make[8]: *** [test-suite.log] Error 1
>>>>>>> make[8]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>> make[7]: *** [check-TESTS] Error 2
>>>>>>> make[7]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>> make[6]: *** [check-am] Error 2
>>>>>>> make[6]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>> make[5]: *** [check-recursive] Error 1
>>>>>>> make[5]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>> make[4]: *** [check-recursive] Error 1
>>>>>>> make[4]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests'
>>>>>>> make[3]: *** [check-recursive] Error 1
>>>>>>> make[3]: Leaving directory
>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests'
>>>>>>> make[2]: *** [check-recursive] Error 1
>>>>>>> make[2]: Leaving directory `/home/lawrence/build/pylith/
>>>>>>> pylith-build'
>>>>>>> make[1]: *** [pylith] Error 2
>>>>>>> make[1]: Leaving directory `/home/lawrence/build/pylith'
>>>>>>> make: *** [installed_pylith] Error 2
>>>>>>>
>>>>>>> I have found this CIG-SHORT post here:
>>>>>>> http://lists.geodynamics.org/pipermail/cig-short/2014-
>>>>>>> August/001833.html
>>>>>>> But it did not help. Attached are the test*.log files along with the
>>>>>>> testsuite.log file
>>>>>>>
>>>>>>> I would appreciate any help in regards to fixing this problem.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> CIG-SHORT mailing list
>>>>>>> CIG-SHORT at geodynamics.org
>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> CIG-SHORT mailing list
>>>>>> CIG-SHORT at geodynamics.org
>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> _______________________________________________
>>>>> CIG-SHORT mailing list
>>>>> CIG-SHORT at geodynamics.org
>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks & Best Regards,
>>>>
>>>> Mamun Miah
>>>> Graduate Research Assistant
>>>> Department of Civil Engineering
>>>> The University of Mississippi
>>>> Oxford, MS 38655
>>>> Phone: (662)-801-1904
>>>> Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
>>>>
>>>> _______________________________________________
>>>> CIG-SHORT mailing list
>>>> CIG-SHORT at geodynamics.org
>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>>
>>
>> --
>> Thanks & Best Regards,
>>
>> Mamun Miah
>> Graduate Research Assistant
>> Department of Civil Engineering
>> The University of Mississippi
>> Oxford, MS 38655
>> Phone: (662)-801-1904
>> Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
--
Thanks & Best Regards,
Mamun Miah
Graduate Research Assistant
Department of Civil Engineering
The University of Mississippi
Oxford, MS 38655
Phone: (662)-801-1904
Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
_______________________________________________
CIG-SHORT mailing list
CIG-SHORT at geodynamics.org
http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20150324/97629823/attachment-0001.html>
More information about the CIG-SHORT
mailing list