[CIG-SHORT] PyLith-build test fails
Mamun Miah
mmiah at go.olemiss.edu
Tue Mar 24 16:01:01 PDT 2015
Please see the attached files. The configure.log file still has IBARRIER on
it since rebuilding PETSc without IBARRIER in the petscconf.h file does not
affect the configure.log file.
On Tue, Mar 24, 2015 at 3:30 PM, Matthew Knepley <knepley at mcs.anl.gov>
wrote:
> Moving this discussion to petsc-maint, since you seem to have a completely
> broken MPI install.
>
> Can you send us $PETSC_DIR/configure.log and make.log.
>
> Background: The run was segfaulting on the IBarrier setup so I had him
> remove that #define and rebuild. Now it is
> faulting on the normal Allreduce. Has anyone seen this before?
>
> Thanks,
>
> Matt
>
> ---------- Forwarded message ----------
> From: Mamun Miah <mmiah at go.olemiss.edu>
> Date: Tue, Mar 24, 2015 at 5:17 PM
> Subject: Re: [CIG-SHORT] PyLith-build test fails
> To: cig-short at geodynamics.org
>
>
> Hi Matt,
>
> After doing 'make clean' first and 'make' next in the PETSC_DIR I was able
> to get rid of the IBARRIER and rebuild PETSc. Then I get the following
> errors after I run a sample problem. I checked this website as suggested in
> the message below: http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind.
>
> <http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind>
> I have tried reconfiguring and building with '--download-mpich' in the
> PETSC_DIR to see if it changes anything but no change at all. I have also
> checked the FAQ (2nd link in the error messages) page for memory corruption
> but could not figure out much from there.
>
> [0]PETSC ERROR: PetscTrFreeDefault() called from
> PetscCommBuildTwoSided_Allreduce() line 179 in
> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
> [0]PETSC ERROR: Block [id=0(64)] at address 0x301e920 is corrupted
> (probably write past end of array)
> [0]PETSC ERROR: Block allocated in PetscCommBuildTwoSided_Allreduce() line
> 168 in /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Memory corruption:
> http://www.mcs.anl.gov/petsc/documentation/installation.html#valgrind
> [0]PETSC ERROR: Corrupted memory
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.5.3-2251-gb193602 GIT
> Date: 2015-02-19 19:20:25 -0600
> [0]PETSC ERROR: /home/lawrence/pylith/bin/mpinemesis on a arch-pylith
> named petaluma by lawrence Tue Mar 24 11:03:51 2015
> [0]PETSC ERROR: Configure options --prefix=/home/lawrence/pylith
> --with-c2html=0 --with-x=0 --with-clanguage=C --with-mpicompilers=1
> --with-debugging=0 --with-shared-libraries=1 --download-chaco=1
> --download-ml=1 --download-fblaslapack=1 --with-hdf5=1
> --with-hdf5-include=/home/lawrence/pylith/include
> --with-hdf5-lib=/home/lawrence/pylith/lib/libhdf5.dylib --LIBS=-lz
> CPPFLAGS="-I/home/lawrence/pylith/include "
> LDFLAGS="-L/home/lawrence/pylith/lib " CFLAGS="-g -O2" CXXFLAGS="-g -O2
> -DMPICH_IGNORE_CXX_SEEK" FCFLAGS="-g -O2"
> PETSC_DIR=/home/lawrence/build/pylith/petsc-pylith PETSC_ARCH=arch-pylith
> [0]PETSC ERROR: #1 PetscTrFreeDefault() line 289 in
> /home/lawrence/build/pylith/petsc-pylith/src/sys/memory/mtr.c
> [0]PETSC ERROR: #2 PetscCommBuildTwoSided_Allreduce() line 179 in
> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
> [0]PETSC ERROR: #3 PetscCommBuildTwoSided() line 237 in
> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
> [0]PETSC ERROR: #4 PetscSFSetUp_Basic() line 552 in
> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/impls/basic/sfbasic.c
> [0]PETSC ERROR: #5 PetscSFSetUp() line 192 in
> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
> [0]PETSC ERROR: #6 PetscSFReduceBegin() line 990 in
> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
> [0]PETSC ERROR: #7 DMLocalToGlobalBegin() line 1893 in
> /home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
> [0]PETSC ERROR: #8 void pylith::topology::Field::complete()() line 710 in
> ../../../pylith-2.1.0/libsrc/pylith/topology/Field.cc
>
>
> On Fri, Mar 20, 2015 at 11:09 AM, Matthew Knepley <knepley at mcs.anl.gov>
> wrote:
>
>> On Fri, Mar 20, 2015 at 11:34 AM, Mamun Miah <mmiah at go.olemiss.edu>
>> wrote:
>>
>>> Dear Matt,
>>>
>>> I have run a sample problem from PyLith tutorial and got the following
>>> PETSc errors:
>>>
>>> [0]PETSC ERROR: #1 PetscCommBuildTwoSided_Ibarrier() line 122 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>> [0]PETSC ERROR: #2 PetscCommBuildTwoSided() line 231 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>> [0]PETSC ERROR: #3 PetscSFSetUp_Basic() line 552 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/impls/basic/sfbasic.c
>>> [0]PETSC ERROR: #4 PetscSFSetUp() line 192 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>>> [0]PETSC ERROR: #5 PetscSFReduceBegin() line 990 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/interface/sf.c
>>> [0]PETSC ERROR: #6 DMLocalToGlobalBegin() line 1893 in
>>> /home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
>>> [0]PETSC ERROR: #7 void pylith::topology::Field::complete()() line 710
>>> in ../../../pylith-2.1.0/libsrc/pylith/topology/Field.cc
>>> Fatal error. Calling MPI_Abort() to abort PyLith application.
>>>
>>> Looks like the IBARRIER is still causing the problem.
>>>
>>
>> This makes it clear that you did not rebuild PETSc. Here is the source:
>>
>>
>> https://bitbucket.org/petsc/petsc/src/6d5951fb15cef55c0a1c902bab2c56b27935b002/src/sys/utils/mpits.c?at=master#cl-86
>>
>> The function listed will not even be compiled if you rebuilt with that
>> #ifdef removed. Please
>> remove that statement from petscconf.h, rebuild PETSc, which will
>> recreate the shared library.
>>
>> Thanks,
>>
>> Matt
>>
>>
>>> On Thu, Mar 19, 2015 at 3:30 PM, Matthew Knepley <knepley at mcs.anl.gov>
>>> wrote:
>>>
>>>> On Thu, Mar 19, 2015 at 4:07 PM, Mamun Miah <mmiah at go.olemiss.edu>
>>>> wrote:
>>>>
>>>>> Deat Matt,
>>>>>
>>>>> I have tried rebuilding PETSc after removing HAVE_MPI_IBARRIER from
>>>>> the petscconf.h file. It did not show any error. Then I went on building in
>>>>> the pylith directory. It shows the message as follows-
>>>>>
>>>>
>>>> This is fine. PyLith does not need to be rebuilt for this change.
>>>>
>>>>
>>>>> lawrence at petaluma:~/build/pylith$ make
>>>>> Making all in bin
>>>>> make[1]: Entering directory `/home/lawrence/build/pylith/bin'
>>>>> make[1]: Nothing to be done for `all'.
>>>>> make[1]: Leaving directory `/home/lawrence/build/pylith/bin'
>>>>> make[1]: Entering directory `/home/lawrence/build/pylith'
>>>>> make[1]: Nothing to be done for `all-am'.
>>>>> make[1]: Leaving directory `/home/lawrence/build/pylith'
>>>>> lawrence at petaluma:~/build/pylith$
>>>>>
>>>>> Also, rebuilding PETSc after removing IBARRIER does not seem to have
>>>>> any effect on the $PETSC_DIR/config.log file.
>>>>>
>>>>
>>>> Also fine, you did not reconfigure.
>>>>
>>>> Please run your example again and see if it works.
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Mamun
>>>>>
>>>>> On Wed, Mar 18, 2015 at 1:57 PM, Matthew Knepley <knepley at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>> On Wed, Mar 18, 2015 at 1:24 PM, Brad Aagaard <baagaard at usgs.gov>
>>>>>> wrote:
>>>>>>
>>>>>>> The test-suite.log file lists the following error:
>>>>>>>
>>>>>>
>>>>>> Yes, it is possible that we are finding the MPI_Ibarrier() function
>>>>>> in our test, but the implementation
>>>>>> actually just crashes. We have seen this before with incomplete MPI
>>>>>> implementations. You can
>>>>>> remove
>>>>>>
>>>>>> HAVE_MPI_IBARRIER
>>>>>>
>>>>>> from
>>>>>>
>>>>>> $PETSC_DIR/$PETSC_ARCH/include/petscconf.h
>>>>>>
>>>>>> and rebuild PETSc using
>>>>>>
>>>>>> cd $PETSC_DIR
>>>>>> make
>>>>>>
>>>>>> Please send
>>>>>>
>>>>>> $PETSC_DIR/configure.log
>>>>>>
>>>>>> so I can look at the test for that function.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>>> test_writeData (TestOutputSolnPoints.TestOutputSolnPoints) ...
>>>>>>> [0]PETSC ERROR: #1 PetscCommBuildTwoSided_Ibarrier() line 122 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>>>>>> [0]PETSC ERROR: #2 PetscCommBuildTwoSided() line 231 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/sys/utils/mpits.c
>>>>>>> [0]PETSC ERROR: #3 PetscSFSetUp_Basic() line 552 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/
>>>>>>> impls/basic/sfbasic.c
>>>>>>> [0]PETSC ERROR: #4 PetscSFSetUp() line 192 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/
>>>>>>> interface/sf.c
>>>>>>> [0]PETSC ERROR: #5 PetscSFBcastBegin() line 923 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/vec/is/sf/
>>>>>>> interface/sf.c
>>>>>>> [0]PETSC ERROR: #6 DMGlobalToLocalBegin() line 1692 in
>>>>>>> /home/lawrence/build/pylith/petsc-pylith/src/dm/interface/dm.c
>>>>>>> [0]PETSC ERROR: #7 void pylith::topology::Field::scatterGlobalToLocal(PetscVec,
>>>>>>> const char*) const() line 1168 in ../../../pylith-2.1.0/libsrc/
>>>>>>> pylith/topology/Field.cc
>>>>>>> ERROR
>>>>>>>
>>>>>>> ============================================================
>>>>>>> ==========
>>>>>>> ERROR: test_writeData (TestOutputSolnPoints.TestOutputSolnPoints)
>>>>>>> ------------------------------------------------------------
>>>>>>> ----------
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/home/lawrence/build/pylith/pylith-2.1.0/unittests/
>>>>>>> pytests/meshio/TestOutputSolnPoints.py", line 194, in test_writeData
>>>>>>> output.writeData(2.0, self.fields)
>>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/
>>>>>>> pylith/meshio/OutputManager.py", line 252, in writeData
>>>>>>> self._appendVertexField(t, field, mesh)
>>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/
>>>>>>> pylith/meshio/OutputManager.py", line 439, in _appendVertexField
>>>>>>> ModuleOutputManager.appendVertexField(self, t, field, mesh)
>>>>>>> File "/home/lawrence/pylith/lib/python2.7/site-packages/pylith/meshio/meshio.py",
>>>>>>> line 300, in appendVertexField
>>>>>>> def appendVertexField(self, *args): return _meshio.OutputManager_appendVertexField(self,
>>>>>>> *args)
>>>>>>> RuntimeError: Error detected while in PETSc function.
>>>>>>>
>>>>>>> This looks like it may be an MPI issue. I don't get this error on
>>>>>>> any of the machines I have tested using OpenMPI and MPICH. What version of
>>>>>>> MPI are you using? How was MPI installed (installer, sys admin, you,
>>>>>>> someone else)?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Brad
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 03/18/2015 11:52 AM, Mamun Miah wrote:
>>>>>>>
>>>>>>>> Dear developers,
>>>>>>>>
>>>>>>>> I have been trying to build PyLith from the source code but
>>>>>>>> apparently
>>>>>>>> reached an impasse. I am getting the following test failure.
>>>>>>>>
>>>>>>>> FAIL: testmeshio.py
>>>>>>>>
>>>>>>>> ============================================================
>>>>>>>> ================
>>>>>>>> Testsuite summary for PyLith 2.1.0
>>>>>>>> ============================================================
>>>>>>>> ================
>>>>>>>> # TOTAL: 3
>>>>>>>> # PASS: 2
>>>>>>>> # SKIP: 0
>>>>>>>> # XFAIL: 0
>>>>>>>> # FAIL: 1
>>>>>>>> # XPASS: 0
>>>>>>>> # ERROR: 0
>>>>>>>> ============================================================
>>>>>>>> ================
>>>>>>>> See unittests/pytests/meshio/test-suite.log
>>>>>>>> Please report to cig-short at geodynamics.org
>>>>>>>> ============================================================
>>>>>>>> ================
>>>>>>>> make[8]: *** [test-suite.log] Error 1
>>>>>>>> make[8]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>>> make[7]: *** [check-TESTS] Error 2
>>>>>>>> make[7]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>>> make[6]: *** [check-am] Error 2
>>>>>>>> make[6]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>>> make[5]: *** [check-recursive] Error 1
>>>>>>>> make[5]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests/meshio'
>>>>>>>> make[4]: *** [check-recursive] Error 1
>>>>>>>> make[4]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests/pytests'
>>>>>>>> make[3]: *** [check-recursive] Error 1
>>>>>>>> make[3]: Leaving directory
>>>>>>>> `/home/lawrence/build/pylith/pylith-build/unittests'
>>>>>>>> make[2]: *** [check-recursive] Error 1
>>>>>>>> make[2]: Leaving directory `/home/lawrence/build/pylith/
>>>>>>>> pylith-build'
>>>>>>>> make[1]: *** [pylith] Error 2
>>>>>>>> make[1]: Leaving directory `/home/lawrence/build/pylith'
>>>>>>>> make: *** [installed_pylith] Error 2
>>>>>>>>
>>>>>>>> I have found this CIG-SHORT post here:
>>>>>>>> http://lists.geodynamics.org/pipermail/cig-short/2014-
>>>>>>>> August/001833.html
>>>>>>>> But it did not help. Attached are the test*.log files along with the
>>>>>>>> testsuite.log file
>>>>>>>>
>>>>>>>> I would appreciate any help in regards to fixing this problem.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> CIG-SHORT mailing list
>>>>>>>> CIG-SHORT at geodynamics.org
>>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> CIG-SHORT mailing list
>>>>>>> CIG-SHORT at geodynamics.org
>>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> _______________________________________________
>>>>>> CIG-SHORT mailing list
>>>>>> CIG-SHORT at geodynamics.org
>>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Thanks & Best Regards,
>>>>>
>>>>> Mamun Miah
>>>>> Graduate Research Assistant
>>>>> Department of Civil Engineering
>>>>> The University of Mississippi
>>>>> Oxford, MS 38655
>>>>> Phone: (662)-801-1904
>>>>> Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
>>>>>
>>>>> _______________________________________________
>>>>> CIG-SHORT mailing list
>>>>> CIG-SHORT at geodynamics.org
>>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> _______________________________________________
>>>> CIG-SHORT mailing list
>>>> CIG-SHORT at geodynamics.org
>>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks & Best Regards,
>>>
>>> Mamun Miah
>>> Graduate Research Assistant
>>> Department of Civil Engineering
>>> The University of Mississippi
>>> Oxford, MS 38655
>>> Phone: (662)-801-1904
>>> Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
>>>
>>> _______________________________________________
>>> CIG-SHORT mailing list
>>> CIG-SHORT at geodynamics.org
>>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> _______________________________________________
>> CIG-SHORT mailing list
>> CIG-SHORT at geodynamics.org
>> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>>
>
>
>
> --
> Thanks & Best Regards,
>
> Mamun Miah
> Graduate Research Assistant
> Department of Civil Engineering
> The University of Mississippi
> Oxford, MS 38655
> Phone: (662)-801-1904
> Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
>
> _______________________________________________
> CIG-SHORT mailing list
> CIG-SHORT at geodynamics.org
> http://lists.geodynamics.org/cgi-bin/mailman/listinfo/cig-short
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
--
Thanks & Best Regards,
Mamun Miah
Graduate Research Assistant
Department of Civil Engineering
The University of Mississippi
Oxford, MS 38655
Phone: (662)-801-1904
Website: http://www.linkedin.com/pub/mamun-miah-eit/21/53/339
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20150324/9c252c25/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 2044966 bytes
Desc: not available
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20150324/9c252c25/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: text/x-log
Size: 84078 bytes
Desc: not available
URL: <http://lists.geodynamics.org/pipermail/cig-short/attachments/20150324/9c252c25/attachment-0003.bin>
More information about the CIG-SHORT
mailing list