Skip to content

Carleton Astronet Admin Log Posts

idl70 and goddard library upgrade

I basically followed the idl64 install instructions already posted in this blog.

Joel had received the idl70 update email from itt, and that contains the instructions on where to download the license file and the tar.gz file containing the install image.

[bduffy 2009-02-04: I’ve put a copy of the relevant license.dat and tar.gz files into /etc/secret/exchange/idl70/]

As root:

cd /usr/local

Because /usr/local/itt already existed and contained idl64, I moved the original /usr/local/itt to itt64 to move it out of the way. I will delete it later after I’m comfortable that the install went smoothly.

mv ./itt ./itt64
mkdir itt
cd itt
mv /root/idl70linux.x86.tar.gz
tar -xvzf idl70linux.x86.tar.gz
cd ./license
cp /etc/secret/idl7-license.txt ./license.dat

I then edited /usr/local/itt/license/license.dat to contain only the license entry for this box.

chmod 644 ./license.dat

cd /usr/local/itt

Then ran the ‘install’ script:

./install

Said ‘y’ to all prompts

Chose option 1) for creating symbolic links and defining the IDL environment. It created several symbolic links.

Finally the install script asked if we wanted to start “DICOM Network Services at boot time”. I answered ‘n’.

Restarted the license mgr by issuing these commands:

/usr/local/itt/idl70/bin/lmdown
/usr/local/itt/idl70/bin/lmgrd -c /usr/local/itt/license/license.dat

NOTE: On Mirzam (but not Sirius) I ran into trouble issuing the lmdown command. Turns out the problem was that the ‘IDL_DIR’ env var was set in root’s .tcshrc file and that changed the scripts behaviour. I removed the setting of the ‘IDL_DIR’ from root’s .tcshrc file and that fixed the problem.

I also removed all the old idl6.3 related /etc/*/*_idl_* files on both Mirzam and Sirius.

Tested that idl70 was installed by issuing the ‘idl’ command at the command prompt. Seemed to work.

To get the idl lm to start at boot time, I edited /etc/rc.d/rc.local and removed this line:

/usr/local/itt/idl64/bin/lmgrd -c /usr/local/itt/license/license.dat

… and added this line:

/usr/local/itt/idl70/bin/lmgrd -c /usr/local/itt/license/license.dat

Made sure all current users ~/.cshrc file contained the correct idl_setup call. They did. Here’s the proper command:

source /usr/local/itt/idl/bin/idl_setup

Then installed the goddard lib:

mkdir /usr/local/itt/idl70/lib/goddard

Downloaded http://idlastro.gsfc.nasa.gov/ftp/astron.tar.gz
to /usr/local/itt/idl70/lib/goddard

cd /usr/local/itt/idl70/lib/goddard
tar -xvzf ./astron.tar.gz

how to run idl procedures (.pro) from val

to compile procedures:
.run filename.pro
for ex, to compile the average.pro procedure i say:

.run average

to execute a given procedure, for example to execute average after compiling the average procedure, i simply type:

average

(See next idlhints post for info on how to run a .batch file)

How to restore files from old 8mm Exabyte tapes

8mm Exabyte backup tapes written between 1994 and 2003 were created with the ufsdump command on a Solaris box. We no longer have the Solaris box, but the Exabyte tape drives are now hooked up to algol (running linux) and can be used to recover the files using restore, the linux port of ufsrestore.

Unlike ufsrestore, restore requires you to specify the blocking factor used to write the files to tape. And, since the dump scripts wrote multiple file partitions to each tape, you also need to tell restore which tape file the partition was written to on the tape. Both the blocking factor and tape file index can be determined by inspecting the the old dump logs stored on algol under /docs/thuban-docs/dumps/

Example:

Joel hands you an Exabyte tape with the label “2/9/95” on it, and asks you to recover a file from the “/usr3” partition that was written to that tape.

1. Login to algol as root

2. cd /docs/thuban-docs/dumps/1995/

3. View the dump log for that date (e.g. ./02.09.95.dmp.gz) using emacs or less (which run gunzip on the fly), and look for ‘/usr3’.

4. Determine the blocking factor used for that tape. Look for a line like: “DUMP: Writing 32 Kilobyte records“, and remember that value (’32 Kilobyte’ in this case). Note that historically the blocking factor changed over time, so always check the blocking factor used for the particular dump in question.

BLOCKING FACTOR = 32KB

UPDATE 2015-06-17: Older tapes (for which there are no dump logs) often used a blocking factor of 10KB.

UPDATE 2015-06-17: If you don’t know the block size, use the commands “mt -f /dev/st0 setblk 0; mt -f /dev/st0 setdensity 0x0” to set the drive to “variable mode”. This gives you lower performance (only 1 block is transferred for each SCSI command, plus Check Condition overhead). If you do this, don’t specify the -b <blocksize> argument when using the restore command.

5. Now determine the index of the tape file the partition you want to recover. Starting at the top of the dump log file, counting from ‘1’, count the number of “DUMP: DUMP IS DONE” lines until you get to the “DUMP: DUMP IS DONE” line for your partition.

In this dump log, partition ‘/usr3’ was written as the 16th tape file to the tape, so:

TAPE FILE INDEX = 16

6. Insert the tape in the Exabyte tape drive

7. Rewind the tape. Currently the Exabyte tape drive is /dev/st0, so issue this command:

mt -f /dev/st0 rewind

8. Tell linux what the tape density is using ‘mt‘. Linux seems to autodetect this, but just to be safe… :

mt -f /dev/st0 setdensity 0x15

If you don’t know the tape density, you can set the density to 0 instead:

mt -f /dev/st0 setdensity 0

9. Tell linux which blocksize to use with ‘mt’. In this case our BLOCKING FACTOR is 32kb, which the ‘mt’ command likes to get as an actual byte count. So we multiply 32 x 1024 to get 32768.

mt -f /dev/st0 setblk 32768

If you don’t know the blocksize, set it to 0 (and don’t specify the -b <blocksize> argument to restore:

mt -f /dev/st0 setblk 0

10. cd to the directory you want to put the recovered files in.

cd /tmp/recovery/1995-02-09

11. Call the ‘restore’ command with the following syntax:

restore -a -i -v -b <blocksize> -s <tape file index> -f /dev/st<tape drive ID>

The above arguments tell ‘restore‘ that we want to recover files in interactive mode (-i), ignore volumes (-a), and run in verbose mode (-v).

Given our BLOCKING FACTOR of 32kb and TAPE FILE INDEX of 16, we issue this command:

restore -a -i -v -b 32 -s 16 -f /dev/st0

restore‘ should now forward the tape to the 16th tape file and then enter interactive mode, which is somewhat ftp-ish.

Once in interactive mode you can use ‘cd‘ and ‘ls‘ to explore the partition. You can specify the files to extract by issuing the ‘add <fname>‘ command. When <fname> is a directory ‘add‘ recursively targets the directory and it’s contents. The ‘ls‘ command prefixes the names of files ‘add‘ed to the extraction list with an ‘*’.

For a full list of commands use the ‘help‘ command.

Once you’ve told ‘recover‘ which files to recover to your local directory with ‘add‘ commands, issue the ‘extract‘ command to initiate the recovery. This can take several minutes, up to an hour in some cases, so go do something else and come back later.

When ‘recover‘ has recovered the files it will ask you:

set owner/mode for '.'? [yn]

Answer ‘n‘.

Here’s a sample interactive recovery session. In this case we restore the directory ‘./applications/p1391/’ from the ‘/usr3’ partition saved to the ‘2/9/95’ dump tape:

[root@algol /tmp/recovery/1995-02-09]$ restore -a -i -v -b 32 -s 16 -f /dev/st0
restore > ls
.:
11.29.dmp applications/ old_lang/ spool/
aipsnewest/ lang/ software/ tmp/

restore > cd applications/
restore > ls
./applications:
book/ games/ hiabs/ psrcat/ tempo/
ftptool/ graphing/ p1391/ starlink/ timing/


restore > add p1391
restore > extract

… [WAIT UP TO AN HOUR] …

set owner/mode for '.'? [yn] n
restore > quit
[root@algol
/tmp/recovery/1995-02-09]$

Kudos to James Fuller for figuring this out.

konqueror (nautilus?) file browser and kwrite (kate?)text editor setup hints

Added by jmw 2020 may 28: Maybe these konqueror and kwrite config hints are still somewhat useful for nautilus and kate???

I find konqueror the best file browser for linux. To customize it a little, for example to get .idl files to open with my favorite text editor, kwrite:

with konqueror running:
settings>>configure konqueror gives some options like what your home dir should be (eg /home/youusername

to make conqueror behave a certain way with a certain kind of file, i made in my home dir (/home/jweisber) a file called dummy.idl

if you select it inside konqueror (one click to highlight it)

go to edit>properties
it should pop open a window labelled “Properties for dummy.idl – Konqueror”
‘way to the right of “type” is an icon of a bunch of tools. push on it.
it should pop open a window labelled “Edit file type application/idl KDEditFileType- Konqueror”
if its not already there, add a filename pattern of *.idl
that means any file ending with that will be treated in the way you are getting ready to set up

under application preference order, enter
kwrite

if you wish, you can also associate a pretty icon with *.idl by clicking on the “?” icon to left of “filename patterns”

click ok and from now on , when you dble-click on any file ending in .idl, it should open it with kwrite (see addition below)

(you may have to entirely quit and restart konqueror for the new properties to take)

–jmw january 2008

——addition feb 2008 – to get kwrite to use “syntax highlighting” (ie to color idl files with a certain color set; fortran etc etc):

open an idl file in kwrite.

in the kwrite toolbar up top, hit settings>configure editor>highlighting

now here is what took me more time than i am willing to admit:

the panel should now be labelled highlighting rules.

the first thing to do is to select a type from the pulldown menu near the top; eg “sources idl ”

then inside properties:file extensions, type *.pro;*.idl;*.batch

inside properties: MIME types, type application/idl

then hit the icon on the far right of this line and get a panel saying “Select the mimetypes you want highlighted using sources/idl syntax highlight rules”

and check the relevant mime types.

(note if the right mime types arent there, see joel)

note that some of these properties do not “take” until you jump out of konqueror and restart it!

Algol can now talk to its SDLT Tape Drive

Algol has never been able to properly communicate with the SDLT tape drive. The symptoms were that you could write/read single small file to/from the tape drive, but an attempt to write a *directory* to it drive would fail with this message in shell:

tar: /dev/st1: Cannot write: Input/output error
tar: Error is not recoverable: exiting now

…and this message in /var/log/messages:

Jan 9 15:20:08 algol kernel: st1: Error with sense data: Current st1: sense key Aborted Command
Jan 9 15:20:08 algol kernel: Additional sense: Synchronous data transfer error

The first thing that James Fuller and I did was to see if the other tape drive, a sun 8mm exabyte drive known to algol as /dev/st0, was working. It worked fine. That indicated that algol’s Adaptec 39160 scsi controller card was as least partly working.

The next thing we did was reboot algol. This wasn’t really planned as part of our attempt to get the tape drive working, but rather to test the newly installed idl7.0 license manager. But it was a good thing we did. During the boot sequence we saw a screen titled “Adaptec SCSI Select: hit Ctrl-A to enter”. Prior to this time we had no idea such a utility existed. It’s basically a BIOS for the scsi controller card.

We hit ctrl-a, entered the utility and poked around. The utility enables you to see, for each of the two “channels” A & B (ports), the devices mapped to device ID’s 0-15. On channel A, only the scsi controller itself is mapped (to device ID 7). On channel B, the exabyte tape drive was mapped to device id 0, and the Quantum SDLT was mapped to device ID 6 and the scsi controlloer to device ID 7.

Each channel screen showed a matrix of the devices and parms for each device. The first such parm was “Initiate Wide Negotiation”. This rang a bell in James’ head — he remembered reading somewhere that this parm, when turned on, could cause problems for some scsi devices. So we turned it off for device 6. Note: making this change had the side effect of changing the communications speed value from 160Mb/s to 40Mb/s.

However, when we exited the utility and allowed the reboot to complete, we could write to/read from the SDLT tape drive.

I verified that it was really working by writing /home to it, reading it back to a temporary location, and then doing a recursive diff (diff -rq /home /root/foodir/home) against the original and the copy read from the tape. There were no diffs, and /home takes up 1.5G so I feel confident that Algol can really talk to the SDLT tape drive now.

idl documentation and help commands from drew

Drew Brisbin gently and helpfully reminded me (Joel) of these three helpful ways of getting online documentation help.

It is useful to remember these three commands to help you find your way in IDL:

1. help, VARIABLE
this will tell you stuff about the variable (is it a string? float? double? is it an array? how big is it?)
So when you did “help, point_lun ” it was trying to find a variable named point_lun. When it didn’t find one, it told you it was undefined.

2. ? IDLCOMMAND
When you hit ? [return] it opens up the IDL help browser. If you stick an IDL command you’re curious about after the ?-mark, it searches for that in the help browser. Very useful for figuring out how to use commands.

3. doc_library, ‘USERCOMMAND’
(remember to use single quotes around USERCOMMAND)
if you’re using a user-defined command (such as readcol, for instance), there wont be any help info in the built in help browser. So instead of using the ?-mark, you have to use doc_library. If the author of the command was courteous enough to include help-like information, it will be displayed when you type this.

Also see Sep 15 post below entitled: IDL path; IDL help; unix inside IDL

Differencing two similar text files

There is a faily brain-dead unix utilitiy for differencing two files:

diff file1 file2

There is a lovely graphical version on linux called kompare. i highly recommend it (jmw).

Its output can be exported as text to send to distant colleagues. Furthermore the following diff command gives similar (text) output – much to be preferred to raw diff:

diff -U 3 -dHrN — file1 file2

(the 3 means it places 3 lines of context around the change to help you visualize it)

–Joel

Thuban reboot issues

Thuban as the server, is more sensitive to rebooting than other machines. Therefore do not reboot thuban unless absolutely necessary.

If it is rebooted, be sure that it has exported all the shared disks for use on other astro net clients. To check that this has successfully happened, log in to another desktop and try cd-ing to various shared disks like /data/psrdata and /docs and so on. If thuban has NOT mounted these disks for sharing, then James Fuller says to enter the following command on thuban:

(as root):# /usr/sbin/exportfs -a

–Joel