Skip to content

Carleton Astronet Admin Log Posts

calculate weighted (mean and SDM) in IDL

Published by jweisber on 2018-03-11

At least for now, function wmeanandsig.pro is in /data/psrdata/rmgalprojs

If you create an array of RM and an equal-sized array of sig_RM (with no ∞ sig_RM’s), then:

wmean_and_sdm=wmeanandsig(RM,sig_RM)

will put the weighted mean and weighted sdm into a two-element array  called wmean_and_sdm

(and the way functions work, you can replace the variable that above i call “wmean_and_sdm”, with

a variable of any name you wish)

Right now there is a “driver” program in the same directory, called, “wmeanandsigDRIVER.pro”  .  It

is a module that tests wmeanandsig.pro  .  I used it extensively to do so, to the point where I can now

call wmeanandsig.pro “ready to go”   .  (Luckily the NIST web pages give sample input/output for weighted

mean and for weighted SDM, which I endeavored to match.)

Caution: In 2018 April, I changed the calculation of the weighted SDM.  Previously, we had used the NIST definition, which changed the data weights so as to achieve reduced χ^2 to 1.  (The NIST webpage gave a formula for wgtd SD but did not note that it included this readjustment. )  In 2018 April, I changed the calculation to be

σ^2_wgtdmean = 1 / [ Σ 1/σ^2_j]. Note that this calculation depends entirely on the σ_j’s of the data points, and nothing else (such as deviation from mean).

–Joel

idl save set query contents

Published by jweisber on 2018-03-02

Here’s how to figure out the contents of a save set.

1. This is the classic way:
restore,’filename.sav’,/verbose  will list the variable names in the saveset.

help, variable_name will then give details on any such variable.

2. This  is the modern (object-oriented) way. Note that it does not require a restore:

[After http://northstar-www.dartmouth.edu/doc/idl/html_6.2/IDL_Savefile.html#wp1034474 ] :

Use the IDL_Savefile object to query the contents of a SAVE file containing data and selectively restore variables.

So first, create a savefile object out of of the .sav file:   sObj = obj_new(‘IDL_Savefile’, ‘p2206.B1913+16.wapp4.53958.0004_shifted.sav’)

Typically, IDL_Savefile::Contents is the first method called on a savefile object; the information retrieved indicates the number of different types of items contained in the file (variables, routines, etc.).

sContents = sObj->Contents()

one type of query of sContents is:  print, sContents.N_var
20

Next, a call to IDL_Savefile::Names is used to obtain the names of items of a specific type.

sNames = sObj ->Names()
print,snames
ACTUAL_NRECS BINSHIFT BW DELFREQ IDLFILE LSRK NBIN NCHAN OBSJULDATE OBSTIME PROFILE_VERSION ROUNDBINSHIFT SCANNO SHIFTONOFFA SHIFTONOFFB
SHIFTTOTA SHIFTTOTB SRC VEL WAPPHOST

Then,  IDL_Savefile::Size can be used to determine the size of a given (named) variable in the file:

size_of_shiftonoffa = sObj->Size(‘shiftonoffA’)
print, size_of_shiftonoffA
1    512   4    512,  which means:

If no keywords are set, SIZE returns a vector of integer type. The first element is equal to the number of dimensions of Expression. This value is zero if Expression is scalar or undefined. The next elements contain the size of each dimension, one element per dimension (none if Expression is scalar or undefined). After the dimension sizes, the last two elements contain the type code (zero if undefined) and the number of elements in Expression, respectively. The type codes are listed at https://www.harrisgeospatial.com/docs/size.html.  (the four here means floating point).

 

–Joel

ATNF PSR catalog from the command line

Published by jweisber on 2018-01-23

It is possible to access the ATNF catalog from the command line. Then various unix commands may be used on the output to create tailored data. The catalog software is now in /data/psrdata/psrcat. There is a README there that gives directions on how to use it.

Note that it accesses a database file on disk which may or may not be the latest version!

Also note that you need not supply the db_file parameter if you desire that it default to the location given in the environment variable PSRCAT_FILE    (you may find the value of that variable by typing

printenv PSRCAT_FILE

if you got your .cshrc from me (Joel) relatively recently, you will probably get
/usr/share/astro/psrhome/psrcat/psrcat.db

I have linked that locataion to the psrcat.db file that is “actually” in /data/psrdata/psrcat

so when i type psrcat -v from that directory, I indeed get the current catalogue version, v1.57:

“Software version: 1.49
“Catalogue version number = 1.57”

I may or may not update the catalog in the future.

Note also that the psrchive suite now accesses the same catalog through a link.  That is, the only file in

/usr/share/astro/psrhome/psrcat is a link to the above /data/psrdata/psrcat/psrcat.db

(OK I also did the same with glitch.db)

IDL license info

Published by jweisber on 2017-11-16

To determine IDL license number on a particular host, I think logging in to idl gives the license number.  But the geeky way is to  look near the top of the following file on a machine hosting an IDL license:

/usr/local/exelis/license/license.dat

(Mirzam is 217420)

–Joel

examples: histogram, creating date+time filename, writing out plot object

Published by jweisber on 2017-06-19

all this and more is in /data/psrdata/usr4/lutzkelker/APJsourcecode/createProbForD.pro      :

creating histogram with coyote cghistoplot and with a simple plot object

decoding systime() into a string of yyyyMOdd-HHmi for filename writeout with date&time embedded

but note that the function call may need to be all lowercase!!!!!Also note that above program now calls the function which resides in the path at /usr/share/astro/idlshare/locallib/getyyyymoddhhmi.pro , and that it is called in main as: mydatimestring=getyyyymoddhhmi(). And furthermore that one can test for existence of a routine in path by typing from command line:  print, file_which(‘getyyyymoddhhmi.pro’)

writing out a postscript from an old style plot procedure and from an already-generated plot object (extremely easy!)

fixing problems with pgplot and cpgplot: building them and linking them to our own pgms

Published by jweisber on 2017-01-22

See updates at bottom from building pgplot and cpgplot on phasshare virtual 64 bit machine

First, recall that there is a pgplot-32bits (obsolete as of 2021 Feb ) and a pgplot (now 64) directory

On Aug 2, 2016, Yuping suggested putting cpgplot.h into /usr/include  .  Note this is NOT a shared directory so will need to be done on whatever host one is working on. If it is not there and one tries to build cpgplot, it will fail.

Another possible problem could occur when trying to link pgplot or cpgplot to a program.

e.g; the (64 bit) linking command

gfortran -o LKBorig LKBorig.o -L/usr/share/astro/psrhome/pgplot -lpgplot -lcpgplot -L/usr/lib64 -lX11 -lm -lstdc++ -lpng -lz

(cpgplot is just a c wrapper for pgplot and Yuping says it was actually pgplot that was damaged or missing.) (But see below 2021 Feb 3 update for a related prob- there we need to type ‘make cpg’  to make libcpgplot.a   . I dont think we use lib….so any more)

–Joel (with help from Yuping)

Update 2021 Feb 3 JMW: Creating pgplot and cpgplot on phasshare 64 bit virtual machine:

First, don’t ignore above advice about certain special files and their placement.

Second, Bruce and Joel could only get cpgplot programs to work by seeing how libcpgplot.a was created and linked , in psrchive  directory(much more below)!

To create a fresh pgplot and cpgplot that work, execute pgplot.csh in the bowels of psrchive:

/usr/share/astro/psrhome/src/psrchive/packages>
./pgplot.csh | tee /usr/share/astro/psrhome/pgplot/pgplot.csh.out
(tee sends the text  output to the named file  in addition to standard output). See a fuller description of what pgplot.csh does below.

(User must move the previous pgplot out of the way before running this script; ie
mv /usr/share/astro/psrhome/src/pgplot/ /usr/share/astro/psrhome/src/pgplot.older/)

(And I worry that maybe I need to move other random carcasses out of the way too.)

Some of the things done by pgplot.csh:
1. create a mother directory to put all necessary pgplot files into:
/usr/share/astro/psrhome/src/pgplot, after first moving aside the current tree there (see above).Actually it creates /usr/share/astro/psrhome/src/ and then will pour everything below it.)
2. Also create /usr/share/astro/psrhome/pgplot/ (I wonder if we should have moved old one out of way here too?)
3. get if necessary and gunzip and de-tar the pgplot 5.2  distribution (!)  and put it into and under newly created mother dir.
4 patch the png routines: Interestingly, there is a file
/usr/share/astro/psrhome/src/psrchive/packages/pndriv.patch which seems to suggest that there might indeed be probs with standard png driver, and it is pulled in by pgplot.csh and does seem to enable the use of png routines with pgplot and I assume cpgplot.
5. fix psrchive.conf
6. enable good output drivers
7. run /usr/share/astro/psrhome/src/pgplot/makemake with appropriate flags, creating master makefile, and ‘make’ -ing it and ‘make cpg’ -ing it.  As far as I can tell, the makefile made from this process is essentially identical to the earlier one(s), except that the earlier ones could not link in the png routines because they were broken, but the current ones are patched by pgplot.csh  (perhaps not in makefile itself but rather in something it calls.)

But for now consider the following:

lots of pgplot source files are in
/usr/share/astro/psrhome/src/pgplot/src/
cpgplot files are in
/usr/share/astro/psrhome/src/pgplot/cpg/  (see cpgplot.doc here for important help and note that pgbind.c ,which is run by ‘make cpg’, actually makes the libcpgplot.a I believe)

Note that the two above directories are not parallel nor do they have any other logical relative location!!!

pgplot executable etc dir is /usr/share/astro/psrhome/pgplot/  One can find things like makefile (manually placed there), libpgplot.a, libcpgplot.a cpgdemo, pgplot.csh.out (the text output of the proper building  of cpgplot (and pgplot) via running pgplot.csh as follows:
/usr/share/astro/psrhome/src/psrchive/packages>
./pgplot.csh | tee /usr/share/astro/psrhome/pgplot.csh.out  (tee sends the output to the file  in addition to   standard output)

Note makefile can be created by makemake in /usr/share/astro/psrhome/pgplot/src/pgplot and in fact running makemake in that dir places makefile there too, so I manually moved it up two directories.

makemake /usr/share/astro/psrhome/src/pgplot/ linux psrchive   [the latter three ‘words’  are the arguments of the makmake command], which erroneously dumped makefile into the above specified dir (upon its own reccomendation) so I had to manually mv it to /usr/share/astro/psrhome/pgplot/     .  In the end I had to make from the above  psrchive version in order successfully to create cpg or at least the png parts of it, and to successfully link cpgdemo.

then ‘make’ makes libpgplot.a  and ‘make cpg’ makes libcpgplot.a  (see steps near bottom of that file)!  There is a problem with png routines though that crashes? (or at least reports errors in) the make cpg  but the psrchive-generated makefile fixes this!

 

More serious problem with cpgplot (fixed by Bruce and Joel 20210204):
If I try to build any c (.c) or c++ (.C)  program and link it with cpgplot up thru Feb 3, it failed because it couldn’t find ANY of the cpgplot routines that the program tries to call. (eg the creation of cpgdemo inside the make cpg part of the makefile, or the pglink64plkc.C script for compiling and linking plkc.C in usr4/lk-lookey). Note that cpgplot is just a c wrapper to fortran pgplot routines and that is what is failing.  Long ago we had this problem and it was fixed by adding a trailing “_” to any fortran routine name.  I see that that is being done now but it doesn’t fix the problem (maybe it exacerbates it?). One example of failure (there are many, one for each unique call of a cpgplot routine):

{phasshare.physics.carleton.edu}:{jweisber}:/data/psrdata/usr4/lk-lookey> pglink64plkc.C
/usr/share/astro/psrhome/pgplot/libcpgplot.a(cpgbeg.o): In function `cpgbeg’:
cpgbeg.c:(.text+0x34): undefined reference to `pgbeg_’

The fix of Feb 4 was to model our 64bit compile and link scripts
(like /usr/share/astro/pglink64cpgdemo.C), on the (more automated) steps that
successfully create it in the makefile.  So now /usr/share/astro/pglink64cpgdemo.C is
my template for making other 64 bit, c compile and link scripts. . . .

I found on Feb 5 that .C (ie capital C) files like usr4/lk-lookey plkc.C can not use the same script as .c (ie lower case c) files because the .C files are c++.  After a (too long) while, I got working a script to do so.

So for a c file <filename.c> one types pglink64.c  <filename>  (with extension stripped off), and

for a c++ file <filename.C> one types pglink64.C  <filename>  (with extension stripped off)

( There may be slight path issues. )  Both scripts are in /usr/share/astro on phasshare

 

spreadsheet csv file reading issues and solutions

Published by jweisber on 2016-11-10

We imported a spreadsheet as csv but couldnt edit it in kate. It said “R/O” [=readonly]  Bruce helped us come up with several fixes:  (Thanks Bruce!)

  1. He installed  on Deneb and Canopus “libreoffice, including ‘calc’, which is an open implementation of Excel.  That’s a nice GUI way to edit .csv files.”    https://wiki.documentfoundation.org/ReleaseNotes/5.2  .  You can launch from the Office menu or via the command line as ‘libreoffice –calc <filename> &’
  2. He found that the problem is that the carriage returns are the wrong format, by using the file command:
    file Summary_edited_JoelNov.csv

        Summary_edited_JoelNov.csv: ISO-8859 text, with very long lines, with CRLF line terminators

    He says that this indicates that the csv has dos-type CRLF, not unixtype.  So he installed dos2unix and ran:

    dos2unix < Summary_edited_JoelNov.csv > Summary_edited_JoelNov.unix.csv    and now
    file Summary_edited_JoelNov.unix.csv

    Summary_edited_JoelNov.unix.csv: ISO-8859 text, with very long lines  ,
    so it is fixed and editable with kate

  3. vim or vi work on the original, even with “bad” CRLF

–Joel

building python 2.7 & related stuff on astronet

Published by admin-joel on 2016-10-05

Yuping:

references:

https://www.digitalocean.com/community/tutorials/how-to-set-up-python-2-7-6-and-3-3-3-on-centos-6-4

https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

——————————————————–

I decided for now to install python 2.7 instead of python 3.0  because that’s what most astronomy python libraries are compatible with (e.g. psrchive). If someone wants to build python 3.0 just substitute the following 2.7.12 with 3.0.* and that should work. Note that I install it in /usr/local/ for each of the computers (i’m not sure what exactly willhappen if multiple hosts try to run the same python) so the following is repeated for each computer we have.

——————————————————

prereqs: make sure you do

sudo yum install tcl tcl-devel tk tk-devel

and check to make sure that all prereqs specified in

https://www.digitalocean.com/community/tutorials/common-python-tools-using-virtualenv-installing-with-pip-and-managing-packages

are satisfied.

install:

First download python from https://www.python.org/ and untar it

>xz -d Python-2.7.12.tar.xz

>tar -xvf Python-2.7.6.tar
>sudo cp -r Python-2.7.6 /usr/local/src/

now we can cd into /usr/local/src/Python-2.7.6 and run

>sudo ./configure --prefix=/usr/local
>sudo make

make will complain about not having the following

>Python build finished, but the necessary bits to build these modules were not found:
>bsddb185           dl                 imageop
>sunaudiodev

These four we can live without (see http://www.kelvinwong.ca/2010/08/02/python-2-7-on-dreamhost/) but if you are missing more than that, google it up.

Now do the following (altinstall is very important otherwise you are gonna break CentOS)

> sudo make altinstall

Now python should be built with executable under /usr/local/bin/python2.7

Now build pip which does package management and virtualenv which lets
you install python library without sudo-ing and makes complicated
dependencies easier to handle (see reference article 2)

 

First you need setuptool because pip depends on it

You can download it from https://pypi.python.org/packages/source/s/setuptools/, untar it and copy it over to /usr/local/src/

then do

> cd setuptools-1.4.2

> sudo /usr/local/bin/python2.7 setup.py install

then you download pip from https://bootstrap.pypa.io/get-pip.py into /usr/local/src/ and do

> sudo /usr/local/bin/python2.7 get-pip.py

and pip should be installed.

Finally you can do the following to get some common libraries installed:

> sudo /usr/local/bin/pip2.7 install numpy scipy matplotlib astropy ipython

you can do this on a single computer, go to some nfs mounted directory (e.g. your home directory) and do

> /usr/local/pip freeze > package_list.txt

which pipes the current installed packages to the file package_list.txt. Then you can hop (ssh) onto other computers and just do

> sudo /usr/local/bin/pip2.7 install -r package_list.txt

And done!

more to come on installing python libraries without being sudo.