Skip to content

Carleton Astronet Admin Log Posts

examples: histogram, creating date+time filename, writing out plot object

Published by jweisber on 2017-06-19

all this and more is in /data/psrdata/usr4/lutzkelker/APJsourcecode/createProbForD.pro      :

creating histogram with coyote cghistoplot and with a simple plot object

decoding systime() into a string of yyyyMOdd-HHmi for filename writeout with date&time embedded

but note that the function call may need to be all lowercase!!!!!Also note that above program now calls the function which resides in the path at /usr/share/astro/idlshare/locallib/getyyyymoddhhmi.pro , and that it is called in main as: mydatimestring=getyyyymoddhhmi(). And furthermore that one can test for existence of a routine in path by typing from command line:  print, file_which(‘getyyyymoddhhmi.pro’)

writing out a postscript from an old style plot procedure and from an already-generated plot object (extremely easy!)

fixing problems with pgplot and cpgplot: building them and linking them to our own pgms

Published by jweisber on 2017-01-22

See updates at bottom from building pgplot and cpgplot on phasshare virtual 64 bit machine

First, recall that there is a pgplot-32bits (obsolete as of 2021 Feb ) and a pgplot (now 64) directory

On Aug 2, 2016, Yuping suggested putting cpgplot.h into /usr/include  .  Note this is NOT a shared directory so will need to be done on whatever host one is working on. If it is not there and one tries to build cpgplot, it will fail.

Another possible problem could occur when trying to link pgplot or cpgplot to a program.

e.g; the (64 bit) linking command

gfortran -o LKBorig LKBorig.o -L/usr/share/astro/psrhome/pgplot -lpgplot -lcpgplot -L/usr/lib64 -lX11 -lm -lstdc++ -lpng -lz

(cpgplot is just a c wrapper for pgplot and Yuping says it was actually pgplot that was damaged or missing.) (But see below 2021 Feb 3 update for a related prob- there we need to type ‘make cpg’  to make libcpgplot.a   . I dont think we use lib….so any more)

–Joel (with help from Yuping)

Update 2021 Feb 3 JMW: Creating pgplot and cpgplot on phasshare 64 bit virtual machine:

First, don’t ignore above advice about certain special files and their placement.

Second, Bruce and Joel could only get cpgplot programs to work by seeing how libcpgplot.a was created and linked , in psrchive  directory(much more below)!

To create a fresh pgplot and cpgplot that work, execute pgplot.csh in the bowels of psrchive:

/usr/share/astro/psrhome/src/psrchive/packages>
./pgplot.csh | tee /usr/share/astro/psrhome/pgplot/pgplot.csh.out
(tee sends the text  output to the named file  in addition to standard output). See a fuller description of what pgplot.csh does below.

(User must move the previous pgplot out of the way before running this script; ie
mv /usr/share/astro/psrhome/src/pgplot/ /usr/share/astro/psrhome/src/pgplot.older/)

(And I worry that maybe I need to move other random carcasses out of the way too.)

Some of the things done by pgplot.csh:
1. create a mother directory to put all necessary pgplot files into:
/usr/share/astro/psrhome/src/pgplot, after first moving aside the current tree there (see above).Actually it creates /usr/share/astro/psrhome/src/ and then will pour everything below it.)
2. Also create /usr/share/astro/psrhome/pgplot/ (I wonder if we should have moved old one out of way here too?)
3. get if necessary and gunzip and de-tar the pgplot 5.2  distribution (!)  and put it into and under newly created mother dir.
4 patch the png routines: Interestingly, there is a file
/usr/share/astro/psrhome/src/psrchive/packages/pndriv.patch which seems to suggest that there might indeed be probs with standard png driver, and it is pulled in by pgplot.csh and does seem to enable the use of png routines with pgplot and I assume cpgplot.
5. fix psrchive.conf
6. enable good output drivers
7. run /usr/share/astro/psrhome/src/pgplot/makemake with appropriate flags, creating master makefile, and ‘make’ -ing it and ‘make cpg’ -ing it.  As far as I can tell, the makefile made from this process is essentially identical to the earlier one(s), except that the earlier ones could not link in the png routines because they were broken, but the current ones are patched by pgplot.csh  (perhaps not in makefile itself but rather in something it calls.)

But for now consider the following:

lots of pgplot source files are in
/usr/share/astro/psrhome/src/pgplot/src/
cpgplot files are in
/usr/share/astro/psrhome/src/pgplot/cpg/  (see cpgplot.doc here for important help and note that pgbind.c ,which is run by ‘make cpg’, actually makes the libcpgplot.a I believe)

Note that the two above directories are not parallel nor do they have any other logical relative location!!!

pgplot executable etc dir is /usr/share/astro/psrhome/pgplot/  One can find things like makefile (manually placed there), libpgplot.a, libcpgplot.a cpgdemo, pgplot.csh.out (the text output of the proper building  of cpgplot (and pgplot) via running pgplot.csh as follows:
/usr/share/astro/psrhome/src/psrchive/packages>
./pgplot.csh | tee /usr/share/astro/psrhome/pgplot.csh.out  (tee sends the output to the file  in addition to   standard output)

Note makefile can be created by makemake in /usr/share/astro/psrhome/pgplot/src/pgplot and in fact running makemake in that dir places makefile there too, so I manually moved it up two directories.

makemake /usr/share/astro/psrhome/src/pgplot/ linux psrchive   [the latter three ‘words’  are the arguments of the makmake command], which erroneously dumped makefile into the above specified dir (upon its own reccomendation) so I had to manually mv it to /usr/share/astro/psrhome/pgplot/     .  In the end I had to make from the above  psrchive version in order successfully to create cpg or at least the png parts of it, and to successfully link cpgdemo.

then ‘make’ makes libpgplot.a  and ‘make cpg’ makes libcpgplot.a  (see steps near bottom of that file)!  There is a problem with png routines though that crashes? (or at least reports errors in) the make cpg  but the psrchive-generated makefile fixes this!

 

More serious problem with cpgplot (fixed by Bruce and Joel 20210204):
If I try to build any c (.c) or c++ (.C)  program and link it with cpgplot up thru Feb 3, it failed because it couldn’t find ANY of the cpgplot routines that the program tries to call. (eg the creation of cpgdemo inside the make cpg part of the makefile, or the pglink64plkc.C script for compiling and linking plkc.C in usr4/lk-lookey). Note that cpgplot is just a c wrapper to fortran pgplot routines and that is what is failing.  Long ago we had this problem and it was fixed by adding a trailing “_” to any fortran routine name.  I see that that is being done now but it doesn’t fix the problem (maybe it exacerbates it?). One example of failure (there are many, one for each unique call of a cpgplot routine):

{phasshare.physics.carleton.edu}:{jweisber}:/data/psrdata/usr4/lk-lookey> pglink64plkc.C
/usr/share/astro/psrhome/pgplot/libcpgplot.a(cpgbeg.o): In function `cpgbeg’:
cpgbeg.c:(.text+0x34): undefined reference to `pgbeg_’

The fix of Feb 4 was to model our 64bit compile and link scripts
(like /usr/share/astro/pglink64cpgdemo.C), on the (more automated) steps that
successfully create it in the makefile.  So now /usr/share/astro/pglink64cpgdemo.C is
my template for making other 64 bit, c compile and link scripts. . . .

I found on Feb 5 that .C (ie capital C) files like usr4/lk-lookey plkc.C can not use the same script as .c (ie lower case c) files because the .C files are c++.  After a (too long) while, I got working a script to do so.

So for a c file <filename.c> one types pglink64.c  <filename>  (with extension stripped off), and

for a c++ file <filename.C> one types pglink64.C  <filename>  (with extension stripped off)

( There may be slight path issues. )  Both scripts are in /usr/share/astro on phasshare

 

spreadsheet csv file reading issues and solutions

Published by jweisber on 2016-11-10

We imported a spreadsheet as csv but couldnt edit it in kate. It said “R/O” [=readonly]  Bruce helped us come up with several fixes:  (Thanks Bruce!)

  1. He installed  on Deneb and Canopus “libreoffice, including ‘calc’, which is an open implementation of Excel.  That’s a nice GUI way to edit .csv files.”    https://wiki.documentfoundation.org/ReleaseNotes/5.2  .  You can launch from the Office menu or via the command line as ‘libreoffice –calc <filename> &’
  2. He found that the problem is that the carriage returns are the wrong format, by using the file command:
    file Summary_edited_JoelNov.csv

        Summary_edited_JoelNov.csv: ISO-8859 text, with very long lines, with CRLF line terminators

    He says that this indicates that the csv has dos-type CRLF, not unixtype.  So he installed dos2unix and ran:

    dos2unix < Summary_edited_JoelNov.csv > Summary_edited_JoelNov.unix.csv    and now
    file Summary_edited_JoelNov.unix.csv

    Summary_edited_JoelNov.unix.csv: ISO-8859 text, with very long lines  ,
    so it is fixed and editable with kate

  3. vim or vi work on the original, even with “bad” CRLF

–Joel

building python 2.7 & related stuff on astronet

Published by admin-joel on 2016-10-05

Yuping:

references:

https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4

https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

——————————————————–

I decided for now to install python 2.7 instead of python 3.0  because that’s what most astronomy python libraries are compatible with (e.g. psrchive). If someone wants to build python 3.0 just substitute the following 2.7.12 with 3.0.* and that should work. Note that I install it in /usr/local/ for each of the computers (i’m not sure what exactly willhappen if multiple hosts try to run the same python) so the following is repeated for each computer we have.

——————————————————

prereqs: make sure you do

sudo yum install tcl tcl-devel tk tk-devel

and check to make sure that all prereqs specified in

https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

are satisfied.

install:

First download python from https://www.python.org/ and untar it

>xz -d Python-2.7.12.tar.xz

>tar -xvf Python-2.7.6.tar
>sudo cp -r Python-2.7.6 /usr/local/src/

now we can cd into /usr/local/src/Python-2.7.6 and run

>sudo ./configure --prefix=/usr/local
>sudo make

make will complain about not having the following

>Python build finished, but the necessary bits to build these modules were not found:
>bsddb185           dl                 imageop
>sunaudiodev

These four we can live without (see http://www.kelvinwong.ca/2010/08/02/python-2-7-on-dreamhost/) but if you are missing more than that, google it up.

Now do the following (altinstall is very important otherwise you are gonna break CentOS)

> sudo make altinstall

Now python should be built with executable under /usr/local/bin/python2.7

Now build pip which does package management and virtualenv which lets
you install python library without sudo-ing and makes complicated
dependencies easier to handle (see reference article 2)

 

First you need setuptool because pip depends on it

You can download it from https://pypi.python.org/packages/source/s/setuptools/, untar it and copy it over to /usr/local/src/

then do

> cd setuptools-1.4.2

> sudo /usr/local/bin/python2.7 setup.py install

then you download pip from https://bootstrap.pypa.io/get-pip.py into /usr/local/src/ and do

> sudo /usr/local/bin/python2.7 get-pip.py

and pip should be installed.

Finally you can do the following to get some common libraries installed:

> sudo /usr/local/bin/pip2.7 install numpy scipy matplotlib astropy ipython

you can do this on a single computer, go to some nfs mounted directory (e.g. your home directory) and do

> /usr/local/pip freeze > package_list.txt

which pipes the current installed packages to the file package_list.txt. Then you can hop (ssh) onto other computers and just do

> sudo /usr/local/bin/pip2.7 install -r package_list.txt

And done!

more to come on installing python libraries without being sudo.

When the IDL license manager halts for no good reason on canopus…

Published by admin-joel on 2016-03-17

Updated by Bruce and Joel July 2019

So apparently the license manager for idl on canopus loses connection to the server from time to time and Bruce found out that the best way to deal with it was just to power cycle it. Yuping encountered the problem again today and tried the following and it worked.

sudo su – (puts one into root and bash shell – must have sudo privileges)

$ cd /usr/local/exelis/idl83

This is where the IDL installation is

$./bin/lmutil lmdown

This will kill all of the license managers.

$ps aux|grep lm

Make sure that all lm’s are killed

$ ./bin/lmgrd > lmgrd-201Y-MM-DD.log

imgrd starts the license managers.

This appears to fix the problem.

-Yuping

On 2016-05-26, joel found that the above did not quitre work without some help.  I had to first create the log file with a sudo touch filename command, and then CONCAT to the log file in the above sh command by adding a second > to the first one.

psrchive build notes

Published by admin-joel on 2016-02-25

PSRCHIVE BUILD NOTES

This file is also located /usr/share/astro/psrhome/PsarchiveBuildNotes.txt
Yuping Huang
02/25/2016

psrchive is built under 64bit and with gfortran as the fortran compiler. Note that they require a 64bit gfortran compiled pgplot. I will start by building psrchive since it well build its dependencies automatically.

—–PSRCHIVE—————
Set $PSRHOME to /usr/share/astro/psrhome/ this is the directory for all the softwares.

Add $PSRHOME/bin to your $PATH variable

$ cd $PSRHOME

Unzip psrcat in the directory (psrcat available at http://www.atnf.csiro.au/people/pulsar/psrcat/download.html). Then add in .cshrc setenv PSRCAT_FILE $PSRHOME/psrcat/psrcat.db

$ mkdir src && cd src

$ git clone git://git.code.sf.net/p/psrchive/code psrchive

$ cd psrchive

$ ./bootstrap

$ ./configure –prefix=$PSRHOME F77=gfortran

Then psrchive will complain about not having required packages. It generetes csh scripts under packages for installation and we will use those

$ cd packages

$./pgplot.csh

Remember to add to .cshrc $PGPLOT_DIR and $PGPLOT_FONTS as the script suggests. In my case it is

setenv PGPLOT_DIR $PSRHOME/pgplot/

setenv PGPLOT_FONTS $PGPLOT_DIR/grfont.dat

You should also add $PGPLOT_DIR to your LD_LIBRARY_PATH in .cshrc.

$./fftw.csh

$./cfitsio.csh

$PSRHOME/x86_64-linux/src/cfitsioTake notes of where the cfitsio source is located. We will need it later. In my case it is under $PSRHOME/src/cfitsio

$source ~/.cshrc

this step validates the changes made to the environmental variables.

$./tempo2.csh

This will install tempo2 but likely pgplot will not be linked and henceforth this is not yet a fully functional tempo2. But it does have the components
that psrchive needs. The tempo2 linked with pgplot will be built seperately later.

You should set your $TEMPO2 variable to the tempo2 runtime directory. In my case it is $PSRHOME/tempo2

$ cd ..

This should bring you back to the psrchive folder where configure is located. Now rerun ./configure. It should not complain about fftw, cfitsio, tempo2 or pgplot

./configure –prefix=$PSRHOME F77=gfortran –with-cfitsio-dir=$PSRHOME/src/cfitsio

If you are still having trouble, look at http://psrchive.sourceforge.net/third/packages.shtml. When I did it it can find everything but cfitsio so I fed in the
–with-cfitsio-dir directory. Now we are ready to build it.

$ make

$ make check

$ make isntall

Done! the binaries should locate in $PSRHOME/bin . You should add it to your $PATH variable.