The FreeBSD Browser
Rich Morin <[email protected]>
FreeBSD is a big system; a complete distribution
contains roughly 100,000 files. Consequently, it's
essentially impossible for anyone to remember which
files are related to each other, and why. So, we all
find ourselves trying to remember (or find out) which
files we should be examining, in order to solve a given
problem. The FreeBSD Browser isn't a panacea, but I
think it is a worthwhile addition to the existing set
of system investigation tools.
Specifically, the browser is a specialized "search
engine" for information on FreeBSD. It can help you
find directories, files, manual pages, papers, and other
useful resources. Let's try solving a typical problem
with and without the browser, to see how it helps.
Using Existing Tools
FreeBSD provides a number of tools for system
investigation. Some of my favorites for interactive use
are apropos, grep,
info, locate, ls,
man, more, whatis,
whereis, and which. If I need to
start writing specialized tools, of course, I start
using things like awk, find,
sed, sh, and (for serious work :-)
perl.
Let's think about how these might be used to find out
which parts of the file system are relevant to a
command, assuming that we already know its name (e.g.,
vi). man vi is a good place to
start; in particular, the last part of a man page tends
to contain some useful pointers:
FILES
/bin/sh
The default user shell.
/etc/vi.exrc
System-wide vi startup file.
...
SEE ALSO
ctags(1), more(3), curses(3), dbopen(3)
The ``Vi Quick Reference'' card.
...
Unfortunately, the list of pointers is seldom as
complete as we might wish. For instance, although the
man page for vi is really very well done, it
says nothing about the /usr/share/vi or
/var/preserve directories. It mentions
some papers, but doesn't indicate where they might be
found on the system.
In short, we have more commands to run. One
possibility, at this point, would be to use the
locate command. Typing
locate vi | more will immediately show us
the names of the desired directories. Unfortunately,
it will also list about 4400 other items, most of which
are totally irrelevant:
/etc/mail/RCS/virtusertable,v
/etc/mail/virtusertable
/etc/mail/virtusertable.db
/etc/mail/virtusertable.sample
/etc/services
/modules/vinum.ko
/sbin/vinum
/stand/help/network_device.hlp.gz
/usr/X11R6/bin/xvidtune
...
In short, locate isn't exactly a
finely-tuned solution. whereis, on the other
hand, is:
% whereis vi
vi: /usr/bin/vi /usr/share/man/man1/vi.1.gz /usr/src/usr.bin/vi
We now know the locations of the executable binary,
man page (troff source version), and source directory.
That sounds interesting; let's go look at the source
code:
% cd /usr/src/usr.bin/vi
% ls -F
Makefile config.h pathnames.h port.h
Uh-oh; there aren't any *.c files. Where's
all the source code? Hmmmm; let's take a look at the
Makefile. As expected, it contains a vital
clue:
SRCDIR= ${.CURDIR}/../../contrib/nvi
Now we can cruise over to the real source
directory:
% cd /usr/src/contrib/nvi
| % ls -F |
| FAQ |
 |
build/ |
|
common/ |
|
ip/ |
|
tcl_api |
| LAYOUT |
|
catalog/ |
|
docs/ |
|
ip_cl/ |
|
tcl_scripts/ |
| LICENSE |
|
cl/ |
|
ex/ |
|
perl_api/ |
|
tk/ |
| README |
|
clib/ |
|
include/ |
|
perl_scripts/ |
|
vi/ |
Because we had the tools we needed and knew how to
use them, this exercise only took us a few minutes.
Still, we had to do a lot of typing and we may have
missed a few things along the way. Not bad, presuming
we didn't miss too much, but we might be able
to do better.
Using the Browser
Let's try using the FreeBSD Browser on the problem.
I'll include some output snippets for hard-copy readers
and the terminally lazy, but the best thing for you to
do is to bring up this link in a separate window:
http://www.cfcl.com/md/fb/42/usr/bin/vi
If the machinery of the Internet (including my poor,
beleaguered server :-) is functioning properly, you
should have the relevant page on your screen in a
matter of seconds. The "Description" section may or
may not contain much of interest, depending on the
item being examined. In some cases, however, it can
be extremely informative.
/usr/bin/vi is an item of type "plain file".
file(1) calls it: "ELF 32-bit LSB executable, Intel 80386, version 1
(FreeBSD), dynamically linked (uses shared libs), stripped".
Looking at the rest of the page, we see all of the
information that our previous exercise gained us, and
more. The following lines tell us about all of the
files and directories we found in our exercise, adding
a bit of useful contextual annotation:
File Tree
Extract
| ? / |
|
the system's root
directory |
| ? : home@ |
|
user home
directories |
| ? : : username |
|
sample user home
directory |
| ? : : : .exrc |
|
rc file for ex(1)
and vi(1) |
| ? : : : .nexrc |
|
rc file for nex(1)
and nvi(1) |
| ? : tmp/ |
|
temporary
files |
| ? : usr/ |
|
non-critical
system files |
| ? : : bin/ |
|
"add-on set" of
user commands |
| ? : : : edit* |
|
executable command
(link) |
| ? : : : ex* |
|
executable command
(link) |
| ? : : : nex* |
|
executable command
(link) |
| ? : : : nvi* |
|
executable command
(link) |
| ? : : : nview* |
|
executable command
(link) |
| ? : : : vi* |
|
executable command
(link) |
| ? : : : view* |
|
executable command
(link) |
| ? : : obj |
|
target tree for
system builds |
| ? : : : usr/ |
|
system build
target for /usr |
| ? : : : : src/ |
|
system build
target for /usr/src |
| ? : : : : : usr.bin/ |
|
build target for
for /usr/bin |
| ? : : : : : : vi/ |
|
build target for
for /usr/src/usr.bin/vi |
| ? : : share/ |
|
architecture-independent
files |
| ? : : : doc/ |
|
documentation
files |
| ? : : : : usd/ |
|
User's Supplementary
Documents |
| ? : : : : : 10.exref/ |
|
"Ex Reference Manual",
etc. |
| ? : : : : : : paper.ascii.gz |
|
ASCII "Ex Reference
Manual" |
| ? : : : : : : summary.ascii.gz |
|
ASCII "Ex/Edit
Command Summary" |
| ? : : : : : 11.edit/ |
|
"Edit: A
Tutorial" |
| ? : : : : : : paper.ascii.gz |
|
ASCII "Edit: A
Tutorial" |
| ? : : : : : 12.vi/ |
|
vi(1) papers |
| ? : : : : : : paper.ascii.gz |
|
ASCII "An Intro. to
Display Editing" |
| ? : : : : : : summary.ascii.gz |
|
ASCII "Ex Quick
Reference" |
| ? : : : : : : vipwh.ascii.gz |
|
ASCII "Vi Command &
Function Ref." |
| ? : : : vi/ |
|
source code for
/usr/bin/vi |
| ? : : : : catalog/ |
|
vi(1) message
catalogs |
| ? : : : : perl/ |
|
vi(1) perl
scripts |
| ? : : : : tcl |
|
vi(1) tcl
scripts |
| ? : : src/ |
|
source code
files |
| ? : : : contrib/ |
|
contributed source
code |
| ? : : : : nvi/ |
|
used by
/usr/src/usr.bin/vi |
| ? : : : usr.bin/ |
|
source code for
/usr/bin |
| ? : : : : vi/ |
|
source code for
/usr/bin/vi |
| ? : var/ |
|
transient system
data |
| ? : : preserve/ |
|
preserved editor
buffers |
| ? : : tmp@ |
|
temporary
files |
| ? : : : vi.recover |
|
vi(1) session
recovery files |
Documentation
| ? csh(1) |
|
tcsh - C shell
with file name completion and command line
editing |
| ? ctags(1) |
|
ctags - create
a tags file |
| ? ed(1) |
|
ed, red - text
editor |
| ? edit(1) |
|
ee - easy
editor |
| ? ee(1) |
|
ee - easy
editor |
| ? ex(1) |
|
ex, vi, view -
text editors |
| ? lp(1) |
|
lp - front-end
to the print spooler |
| ? nex(1) |
|
ex, vi, view -
text editors |
| ? nvi(1) |
|
ex, vi, view -
text editors |
| ? nview(1) |
|
ex, vi, view -
text editors |
| ? red(1) |
|
ed, red - text
editor |
| ? ree(1) |
|
ee - easy
editor |
| ? rs(1) |
|
rs - reshape
a data array |
| ? tcsh(1) |
|
tcsh - C shell
with file name completion and command line
editing |
| ? vi(1) |
|
ex, vi, view -
text editors |
| ? view(1) |
|
ex, vi, view -
text editors |
| ? re_comp(1) |
|
re_comp,
re_exec - regular expression handler |
| ? re_comp(1) |
|
re_comp, re_exec -
regular expression handler |
| ? re_exec(1) |
|
re_comp, re_exec -
regular expression handler |
|
| ? regexp(3) |
|
regcomp, regexec,
regsub, regerror - regular expression handlers |
| ? regsub(3) |
|
regcomp, regexec,
regsub, regerror - regular expression handlers |
|
| ? termcap(5) |
|
termcap - terminal
capability database |
| ? terminfo(5) |
|
terminfo - terminal
capability database |
|
| ? environ(7) |
|
environ - user
environment |
The browser also tells us about some items that our
manual process missed, including some hard-linked
commands (e.g., /usr/bin/nvi), the target
directory for the build
(/usr/obj/usr/src/usr.bin/vi), some other
intriguing directories (/usr/share/vi and
/var/preserve), and a slew of man pages.
Finally, it gives us the exact location of the papers
that the man page mentioned, in case we want to take a
look at them.
The hard link information tells us that several
commands share the same executable image. This is
useful information, to be sure, but it also serves
another purpose. If we had started our "traditional"
inquiry with the "ex" command,
whereis wouldn't have been nearly as
helpful:
% whereis ex
ex: /usr/bin/ex /usr/share/man/man1/ex.1.gz /usr/src/contrib/nvi/ex
Although whereis found
.../contrib/nvi/ex, it failed to find
.../usr.bin/vi. Consequently, we would
have had to do a bit more looking to find the
"official" source directory. Because the browser is
able to integrate information from a variety of
sources, it tends to sail past gaps in the linkage
information. A reference to a man page or a
file might never be seen in the traditional method,
but the browser uses it automatically.
The browser also eliminates a great deal of typing.
Although it isn't obvious from the samples above, each
listed item (whether a file, directory, or man page)
is a link to another browser page. So, if your
curiosity is aroused by an entry, a simple mouse click
will bring instant gratification.
In addition, each item's line starts with a
? link, leading to a "linkage trail" page.
This page details the links that were followed to get
from the original item to the item in question. Aside
from satisfying the user's curiosity, this page can
allow the user to analyze and report linkage errors.
Fancier Tricks
Going back up to the top of the page, we see a few
other items of interest. The "Target" form already
contains the current path (or manual page) name, as a
useful starting point. Edit this as desired and hit
"return"; you'll be taken to the appropriate page.
Above the "Target" form, you should see a few more
links. There may well be more over time, but for now,
there are "search", help", and "comment" links.
The "search" link brings up a separate window for
keyword-based searching. The searching algorithm is
roughly the same as that used by whatis(1),
but multiple keywords may be required. And, of
course, the returned man page names are clickable
links.
The "help" link brings up a separate window, with some
fairly minimal online help information and a link to
some background information on the browser and its
motivations. You may find this interesting to review,
but this article covers many of the same points in
rather more detail.
The "comment" link is just a mailto, with
the subject line already filled in. Use this if a
page contains an error, is missing some information
that it should contain, or whatever. The browser's
current "knowledge base" is my own work, based on
system documentation, some investigation, and my own
(all too fallible) memory. As users tell me about
problems they notice, I'll be able to improve the
browser's annotations, link information, and built-in
rules.
While we're on the subject of "built-in rules", let's
consider the question of /dev. Everyone
knows what a mess it is (and devfs actually
promises to clean it up quite a bit), but finding the
relevant information for a given device node can still
be an ordeal. For starters, there's the problem of
finding the device's "base name", so we can bring up
the relevant man page.
With hundreds of device families and dozens of
name-formation conventions, even looking up the man
page for a given device (e.g., /dev/esa0)
can be a hassle. Figuring out the implications of a
given device name and finding the driver sources, of
course, is even harder. So, what can the FreeBSD
Browser tell us about it?
Description
/dev/esa0 is a member of the sa ((SCSI) Sequential Access devices)
family. This device node has the attributes "eject on close, unit 0".
Related device nodes have names that match the Perl regular expressions
/^[en]?r?sa[0-9]+$/ and /^[en]?r?sa[0-9]+\.[0-9]+$/.
File Tree
Extract
| ? / |
|
the system's root
directory |
| ? : dev/ |
|
device nodes |
| ? : : esa0 |
|
device node |
| ? : : sa0 |
|
sample device
node |
| ? : : src/ |
|
source code
files |
| ? : : : sys/ |
|
source code
files for the kernel |
| ? : : : : cam/ |
|
CAM (Common Access
Method) sources |
| ? : : : : : scsi/ |
|
SCSI CAM (Common
Access Method) sources |
| ? : : : : : : scsi_sa.c |
|
source code for
/dev/esa0 |
| ? : : : : : : scsi_sa.h |
|
source code for
/dev/esa0 |
Documentation
This time, the "Description" is a bit more useful. It
tells us the nature of the device, which modes are
being specified, and which other device nodes are
related to it. More precisely, it lists some Perl
regular expressions that will match the names of
related devices. Not everyone reads these things
easily, however, so I have added a "regular
expressions" link. This link brings up a page that
explains each of the regular expressions, in
excruciating detail:
The Perl regular expression
/^[en]?r?sa[0-9]+$/ can be expanded as:
| /^ |
|
# start of
text string |
| [en]? |
|
# possibly the
letter "e" or "n" |
| r? |
|
# possibly the
letter "r" |
| sa |
|
# the device's
base name |
| [0-9]+ |
|
# one or more
decimal digits |
| $/x |
|
# end of text
string |
The Perl regular expression
/^[en]?r?sa[0-9]+\.[0-9]+$/ can be expanded as:
| /^ |
|
# start of
text string |
| [en]? |
|
# possibly the
letter "e" or "n" |
| r? |
|
# possibly the
letter "r" |
| sa |
|
# the device's
base name |
| [0-9]+ |
|
# one or more
decimal digits (unit number) |
| \. |
|
# the special
character "." (period) |
| [0-9]+ |
|
# one or more
decimal digits (mode bits) |
| $/x |
|
# end of text
string |
The prefix letters "e" and "n"
stand, respectively, for "eject on close" and "no rewind
on close". The prefix letter "r" stands for "raw mode",
but it is ignored; this device is always accessed in raw
mode. The first set of digits is used to specify device
units; The second set (mode bits) provides an alternative
way to encode the "e" and "r" information.
Limitations
The browser occasionally brings up irrelevant material.
This may be the result of bad input data (e.g., a
misleading "SEE ALSO" reference), a collision in the
man page "name space", or a poorly-implemented linkage
rule.
A recent version of the browser, for instance, picked
up a false trail from sa(4) to
/sbin/sa (a system accounting program).
Following this lead, it brought in a number of other
irrelevant items (e.g., sa(8),
acct(2), and the source and build
directories for /sbin/sa).
Once a false trail has been discovered (and reported!),
it can usually be resolved with little effort.
Finding them all, however, will be quite a challenge.
The browser's database contains about 100,000 nodes
and a million or so links, with more on the way. So,
there is a lot of room for things to go awry. A few
"false positives" can be tolerated, however, if the
alternative is to lose out on real information.
The browser really should be able to display the
contents of items (e.g., directories, files, and man
pages). The code for this isn't particularly hard (in
fact, I already have most of it on hand), but the
resource drain might well swamp my poor server, to say
nothing of my Internet connection. I hope, however,
to resolve this limitation fairly soon.
Nor does the browser cover the Ports Collection,
systems other than FreeBSD, or (generally) information
about the contents of files. These are, however,
goals of the Meta
Project. In fact, the FreeBSD Browser is only the
current version of a much larger "Meta Demo". That,
in turn, is only a minor subset of the full Meta
proposal. Still, the FreeBSD Browser is a reasonable
starting point; I hope you find it useful...
- Rich
Return to the
March 2001 Issue
