Remote, Secure Zone Console Login

I have heard from a number of customers that folks would like remote login to zone consoles. In particular, they would rather not give out logins to the global zone in order to allow zone logins. (Really: I don’t spend all of my time on the zones console…).

Fortunately, we can handle this in a nice way already. (Disclaimer: Please
note that as stated by the script, the following techniques have not been subject to
a rigorous security audit. I believe this technique to be sound, but neither I nor
Sun warrant it to be so.

To start, we’ll
add a user account to /etc/passwd for each zone we want to set up this way:

# cat >> /etc/passwd

# pwconv
# passwd z1
New Password: xxxyyy
Re-enter new Password: xxxyyy
passwd: password successfully changed for z1

In this case, the zone name is xanadu-z1 and we’ve picked a nice large UID and group ID. You could
use whatever you like (but not a UID in use for something else! and never 0); you’ll want a separate UID for each zone. In this case, /opt/extras/zoneshell
is set as the z1 user’s shell. We picked ‘z1’ as the account name because UNIX
systems are typically limited to 8 letter account names (LOGNAME_MAX); since xanadu-z1 is 9 characters long (and zone names may be up to 64 characters long), we need to pick a convention to shorten things.

The zoneshell script is here; the script itself
is very simple: it looks up the entry in /etc/passwd and executes zlogin -C for the zone named in the GECOS field.

Finally, we need to give the z1 account the ability to run zlogin; we do that by modifying
the RBAC attributes for the z1 user.

# cat >> /etc/user_attr
z1::::profiles=Zone Management

So, here’s what it looks like:

$ ssh -l z1 xanadu
Last login: Tue Jan 25 13:54:01 2005 from xxx
warning: using experimental, unsupported 'zoneshell'
[Connected to zone 'xanadu-z1' console]

I’d appreciate any feedback on whether this is helpful, or not!

To reiterate: this code is experimental, and has not been audited for its security characteristics. Use of this script is AT YOUR OWN RISK. Please use this as an example, from which you could derive your own implementation.

Clearing up confusion about zlogin, zones, consoles, and terminal types

Thanks to bloglines’ nice search feed feature, I found
this thread on the Solaris x86 Yahoo group

Phillip asks why, when he issues a zlogin -C to a zone, it asks him which terminal
type he’d like to use. For those who might not have seen zlogin before, it’s a tool patterned
on the syntax of rlogin and ssh; one uses it to enter a zone from the global
zone. The “regular” way one would use it is as follows:

$ zlogin myzone

This will insert you into the zone with an appropriate subset (including $TERM) of
your environment propagated from the global zone. So, if you are using an xterm and your $TERM
is “xterm”, then that will be propagated correctly into the zone. This is all implemented using
pseudo-terminals (the same things used to make telnet, ssh, etc. do what they do); they are
pretty easy to deal with– when you need one, you create it from nothing, then start some
processes which are connected to it in some fashion. You have full control of the process environment. In this mode, zlogin will never ask you what terminal type you have; if $TERM is unset in your global zone shell, it will either be
unset, or default to something like dumb inside the zone,
depending upon your shell.

Zones also possess a virtual console, which can be accessed using the zlogin -C command.
And this is where Phillip is having problems.
A console is fundamentally different from a pseudo-terminal. While a pseudo-terminal vanishes once you
stop using it, a console (real or virtual) keeps its state; you can connect to and disconnect from
it at any time. Users familiar with using the tip(1) command or other serial console systems
know that they must often tweak some settings after attaching to a console. Think of the console as
television– the programs are always playing, regardless of whether the set is on or not; you
can choose to watch the set or not by turning it on (i.e. connecting to the console).

Phillip recalls having already answered that question when he installed
the system as a whole. In a subsequent post he is
more critical
, since it isn’t intuitive why we ask for this information again.
Since this is my work, hopefully I can show why this isn’t “sloppy” as Phillip
asserts, but rather an unavoidable artifact of the way UNIX consoles function.

To understand this, we need to turn to another important distinction: the terminal type of the system’s
console should usually be set to reflect the kind of hardware which comprises the physical system console.
On Sun’s SPARC boxes, this is sun and on x86 we have sun-color. This is important,
because these terminal types are pretty much incompatible with, for example, the xterm terminal type.

On the other hand, if a machine’s console is instead set to be one of
its serial ports, and is accessed over a tip line, then the default terminal type is usually set to
something fairly benign like xterms (xterm-small) or vt100 or the like– but
this setting must be made by the administrator
because there is no protocol for serially connected
terminals to identify their terminal type, a limitation of the hardware.

Zones emulates the latter sort of connection– a zone console is analogous to a serially connected
tip line. At one end is your terminal, the type of which is not automatically known to the console
at the other end. It is probably an xterm (xterms), a gnome-terminal, a dtterm, or the like.
It might also be a vt220, a wyse or any of hundreds of others. So,
just as we do at first system boot (if we can’t work out what type of terminal we are connected to by
querying the openprom device), we query the user the first time the zone is booted. After that, we’ll
remember this setting. I suppose that we could have just defaulted to something such as ‘vt100’ but that
also seems unfriendly; the sysid tools (the stuff that asks you for your hostname, timezone, etc.) make extensive use of curses, which tends to spam your terminal with garbage if it’s idea of your terminal type doesn’t match
your terminal hardware (or emulator). We certainly can’t default to the system’s default setting, since that
is highly unlikely to be compatible with your window system terminals; if the zone operates as though
the terminal is sun and you are using an xterm, you won’t be pleased.

It’s also worth mentioning that you can automate away all of these first-zone-boot questions by
employing an /etc/sysidcfg file.

Next up– how do you change the terminal type if you’ve made a mistake during the sysid configuration? You’ll
know this happened if your screen is filled with gobbledygook characters when you’d normally see the “what is
your hostname?” question. It’s nice that you have the non-console zlogin available when you encounter situations
like this. To repair things, log off of the zone console, and run:

# zlogin myzone /usr/sbin/sys-unconfig

This will halt the zone after blanking it. Boot the zone back up, log onto the console, and start again.

[Update: It’s not the case that after you specify the terminal type for the sysid tools,
this will automatically become the console terminal type; arguably, this would be good, but it also
doesn’t match the behavior of earlier Solaris releases. We’ll take a look. See the tip below for
how to set your console’s terminal type]

What if you changed your mind about what the default terminal type of the console ought to be?
The classic “big hammer” method is to simply run the sys-unconfig utility; this has the
downside of pretty much blanking your system’s networking configuration, but it is effective.

In older versions of Solaris, you can also edit the ttymon line in /etc/inittab.
Starting in recent builds of Solaris 10, all of this is controlled by SMF, the new Service Management
Facility; as a result, changing the terminal is pretty simple. To check your current setting:

$ svcprop -p ttymon/terminal_type system/console-login

To see all of the ttymon family of properties, issue:

$ svcprop -p ttymon system/console-login
ttymon/device astring /dev/console
ttymon/label astring console
ttymon/modules astring ldterm,ttcompat
ttymon/nohangup boolean true
ttymon/prompt astring \\`uname\\ -n\\`\\ console\\ login:
ttymon/timeout count 0
ttymon/terminal_type astring sun

And to change your console terminal type (as root):

# svccfg -s system/console-login 'setprop ttymon/terminal_type = sun'

So now we have a supported, upgrade-safe way to change all of the elements of the console’s
configuration. Sweet!

More on Bootchart for Solaris

So Eric Let the cat out of the bag
on our reworked BootChart for Solaris. Eric
posted one image
; here is another: the bootup
of a single Solaris Zone
. I’m pretty happy with this, as we boot in only about 7 seconds.

This was an interesting experience because Eric and I had not previously worked together very closely.
I had a great time doing this, and because Eric immediately stuck everything into a
Teamware workspace, we were able to work
simultaneously. Eric and I both worked on the D scripting, and somehow a joke about “wouldn’t it
be nice if the script output XML?” turned into our strategy. This turned out to be a good decision,
as I hate writing parsers; instead we just let SAX do the work. We were able to maintain the split of having boot-log generation in one self-contained
component, and the log processing into another. Because the XML logs are in an easily parsed format
(as opposed to parsing the output of top and iostat), they can be useful to
anyone doing boot analysis on Solaris. We’ve already had some such requests. I’m sure Eric
will have more to say about the implementation so I’ll leave it to him, except to say that some
of the visual design changes can be blamed on me, taking inspiration from
Edward Tufte’s work.

Something else which fell out of this experience is that it’s easy to use the log gatherer on any collection
of processes which start up (as we saw in the zones example, above). We hope that this will be helpful
in highlighting performance wins in the startup of complex ISV programs.

Following the experience of Linux developers, we’ve also found a series of bugs in Solaris with this tool.
Let’s start with an easy one, and the first one I found. The bug is visible in this chart from xanadu,
Shuttle SN45G4.
we don’t have support (sadly) for the on-board ethernet on this box, I had inserted another network card
(I won’t name the vendor, as I don’t want to put them on the spot). If you look carefully at
this bootchart, you’ll see that the ifconfig process is spending a lot of time
on the CPU. What’s up with that? A brief investigation with DTrace made it clear that
the driver had code like the following (shown here reduced as pseudo-code) in the
attach(9E) codepath (attach is the process by which a driver begins controlling a
hardware device):

for (i := 0 to auto_negotiation_timeout) {
if auto_negotiation_complete()
return success;

Which all looks fine except that wait_milliseconds() (a function defined by the driver) is a wrapper around
(“busy-wait for specified interval”). Busy is of course the problem. drv_usecwait
is really more about waiting for short intervals for various information to become ready in
various hardware registers. Busy-waiting 100 milliseconds at a time is practically forever, and
ties up the CPU just spinning in a loop. The authors almost certainly meant to
use delay(9F). I filed
a bug, and hopefully we’ll have it fixed soon (since this driver comes to us from a third party,
they request that we let them make the code changes). Fun, eh?

Another two issues we spotted concern inetd, which has been rewritten from scratch in
Solaris 10; it is now a delegated restarter, which basically means that it take some of
its direction from the system’s master restarter (svc.startd). The behavior we noticed
sticks out on any of the boot charts, including the
zone boot chart
mentioned above: inetd is
starting a lot of very short-lived ksh processes. Why? When I first spotted this,
I used DTrace to work out the answer, as follows:

# dtrace -n 'proc:::create/execname=="inetd"/{ustack();}'
restart inetd
0  12188                     cfork:create`__fork1+0x7`wordexp+0x16f

Ahh, so we stumble upon an (embarrassing) implementation artifact of libc— it uses
ksh to help it implement the libc routine wordexp(3c). So, every time inetd needs
to wordexp() something, we wind up running a ksh. We can also see that this is not severely
impacting performance, but we would like to get this fixed. Personally, I’d like to see
wordexp() fixed to not rely upon ksh at all.

Another somewhat more subtle issue is something that SMF engineers like Liane are still looking at. It’s also visible in
the zone boot chart. It appears that some services (such as nfs/client) are delayed
in coming on-line because it is taking the startd/inetd cabal a while to
mark their dependent services as online, even though that shouldn’t really entail much work.
We can see this as follows:

$ svcs -l nfs/client
fmri         svc:/network/nfs/client:default
name         NFS client service
dependency   optional_all/none svc:/network/rpc/keyserv (online)
dependency optional_all/none svc:/network/rpc/gss (online)
dependency require_all/refresh svc:/milestone/name-services (online) $ svcs -l network/rpc/gss fmri svc:/network/rpc/gss:default name Generic Security Service enabled true state online next_state none state_time Fri Jan 07 18:22:32 2005
restarter svc:/network/inetd:default
dependency require_all/restart svc:/network/rpc/bind (online) dependency optional_all/none svc:/network/rpc/keyserv (online)

Since nfs/client depends upon GSS, it can’t be started by startd
until GSS is online.
Liane and Jonathan
have offered up some theories about why this is happening, but we’ve all been engaged on
higher priority work, and so we haven’t had much time yet to dig deeper.
This serialization point appears to be costing us roughly 1 full second on boot, so
it’s something we need to look at further. Have a great weekend folks!

Whoohoo! gtkam & USB on Solaris 10

With a little help from Frits, I got my Canon Powershot S45 digital camera connected via USB to my Toshiba Tecra M1, all running
Solaris 10, build 74. The most important steps are:

# add_drv -i '"device_id"' ugen

See the ugen(7D) man page for more details; there is also some information and hints in /usr/sfw/share/doc/libusb/libusb.txt. The key trick I discovered is that it is critical to get the device ID correct. To figure
that out, look in your prtconf -v output. You need to find the compatible property.
In my case, for my camera, this was as follows (see also the full screen shot).

name='compatible' type=string items=9
value='usb4a9,306c.1' + 'usb4a9,306c' + 'usb4a9,classff.ff.ff' +
'usb4a9.classff.ff' + 'usb4a9.classff' + 'usb,classff.ff.ff' +
'usb,classff.ff' + 'usb,classff' + 'usb,device'

For whatever reason, you need to get the binding to the most specific device (i.e., the first
in the list). So for me:

# add_drv -m '\* 0666 root sys' -i '"usb4a9,306c.1"' ugen

Also make sure to get the quoting exactly right, as I specify above. Now, unplug your camera,
and plug it back in. Check with prtconf -D and make sure ugen has bound to the device.
If not, you might need to just reboot. Anyway, double check the device permissions, and correct them to
taste. If you’ve followed along they should be 0666 at this point.
The device appears as /dev/usb/id/0/... (0 is the USB ‘interface’ number). Just chmod
the device nodes if they’re not right.

At this point (I think) you should be able to run gphoto2. In build 74 and later,
this appears in /usr/demo/jds/bin, since it’s not officially supported by Sun.

$ /usr/demo/jds/bin/gphoto2 --auto-detect
Model                          Port
Canon PowerShot S45 (normal mode) usb:

If that doesn’t work, try the --debug flag. Next up, try /usr/demo/jds/bin/gtkam,
and see if it works. Good luck! Frits also mentioned that the USB folks are working on ways of
making this process a little less arduous in the future. See also the new USB on Solaris FAQ.

P.S.: the pics are from my friends Dave and Amy’s recent wedding.

P.P.S: Clicking on the thumbnail will also get you a nice shot of my desktop setup on S10_74.

Three Cheers for Firefox’s “Web Developer”

Wow. If you ever have to spend time writing HTML or CSS, have I got a treat for you. Check out the Web Developer Extension written by Chris Pederick.

This thing is great because it lets you reveal the structure of your page, including anchors, blocks, deprecated elements, image meta-data, edit the CSS of a page, validate HTML, CSS, and other characteristics, and make good choices about accessability. It also includes handy buttons which take you to the relevant standards. It’s thoughtfully organized, and has attractive, colorful icons.

Probably the two features which really made my jaw drop are Display Topographic Information which re-renders your page to show you the level of nesting using topographic-map style colors; and View Style Information, which converts your mouse cursor to a crosshair. As you mouse over different page elements,
it shows you the “path” to the object. So on this page, when you mouse over The View from the Moon in the title, you see:

html > body > table > tbody > tr > td.title > div > div.website_title

in the status bar. Clicking on the element pops up a new tab which contains the style information
pertinent to that element. So if you ever wondered “how did they do that?” or “what do I have to do to the CSS to affect this element?” it’s now trivial to figure out.

Solaris Challenge Competiton Update

I posted a while back about our
Solaris 10 Challenge Competition
. Simply put: do something cool with Zones or DTrace, and compete to win a HDTV, ACER Ferrari laptop or
Apple iPod. I got an update from the challenge-master today about the popularity
of the contest. I had expected maybe a dozen entries in each of the four categories. Well, no. Here
are the number of entrants as of today:

  • DTrace Creative Challenge: 44 entries
  • DTrace Performance Challenge: 117 entries
  • Solaris Containers Creative Challenge: 64 entries
  • Solaris Containers Utilization Challenge: 42 entries

For a grand total of 267 contest entries from 39 countries to date! This rocks. And the contest doesn’t close until
December 24, so there’s still time to enter!

What’s New in Solaris Express 11/04 (Build 72)

Solaris 10’s Build 72 release happened today. This release contains
a relatively small number of enhancements, although the AMD64 64-bit kernel
and inclusion of StarOffice are certainly not small changes.
More generally, this build represents a
substantial increase in “polished” fit and finish over Build 69,
which was a huge feature build.
Increasingly our focus is on bug fixing and prepping for the FCS release
Here is a summary of the changes likely to affect you:

x86 Platform

  • The 64-bit kernel is available. On AMD64-architecture systems
    (including Opteron and Nocona) the 64-bit kernel boots by default
    on fresh installations. On upgrades, the 32-bit kernel is booted.
    To boot the 64-bit kernel when it is not the default, use the
    boot file ‘kernel/amd64/unix’. 64-bit versions are also available
    for many libraries. Related ‘blog entries:

  • IPMI support
    is available for lights-out management on x86 systems which
    support it. ipmitool
    is available in /usr/sfw/bin/ipmitool.

  • The Solaris FDisk partition ID has changed. The old identifier
    was 130 (0x82) and the new identifier is 191 (0xbf). Solaris will
    still only allow one partition id to be active. If you need,
    you can use the fdisk command to switch the partition id back and
    forth from ‘Solaris’ (0x82) to ‘Solaris2’ (0xbf). This should solve
    the problem in which
    Linux decides to swap over the Solaris partition
    In the next build, upgrade will automatically change the Solaris
    partitions to the new partition IDs, but as far as I can tell, that’s
    not been finished yet. SPARC systems are not affected.


  • Staroffice and
    Starsuite have been
    integrated into the base product.

  • kdmconfig is now more intelligent, and can help you switch back and
    forth between Xorg and Xsun.

  • Cool new logo integrated into dtlogin!

Service Management Facility

  • Method contexts may now be modified to include changes to environment
    variables. For example, this makes it possible to LD_PRELOAD a library in
    front of an application, without changing its start method.

    # svcprop /network/ssh | grep environment
    start/environment astring UMEM_DEBUG=default

    This facility applies to both svc.startd(1m) and inetd(1m).

  • The system/console-login service has been enhanced with properties which
    allow you to persistently set terminal type and other ttymon parameters.


  • Apache 2.x (2.0.52) is now
    available in /usr/apache2; among other enhancements, Apache 2.x includes
    IPv6 support. Refer
    to this
    for more details. It’s also available as an SMF service,
    so that you can simply svcadm enable apache2 to turn it on (although
    you’ll also need to configure it…).

  • A driver for the
    S2io Xframe 10GB NIC
    has been integrated for all platforms. See xge(7D).

  • A new prompt for configuring NFSv4 is emitted on first system boot
    after a fresh install or sys-unconfig. The prompt allows the user
    to either accept the default domain name derived by nfsmapid(1m)
    for use by NFSv4 or override this domain name by explicitly specifying
    a different domainname.
    Jumpstart install users should note that these prompts can be
    suppressed by having the finish script of the sysidcfg profile
    calling a script such as ‘set_nfs4_domain’, provided in the
    Misc/jumpstart_sample directory of the install image.


  • The sshd daemon now supports a form of privilege separation similar
    to that of OpenSSH, but only after authentication completes. This feature
    is always enabled.


  • /usr/demo/dtrace is populated with all of the examples from the
    DTrace documentation.

  • Webmin has been integrated, and is available
    in /usr/sfw; running /usr/sfw/lib/webmin/ will set it up
    and install it as an SMF service. This includes SSL and PAM integration.

LISA ’04: Solaris Zones: Operating System Support for Consolidating Commercial Workloads

Here is a copy of our LISA ’04 paper, entitled Solaris Zones: Operating System Support for Consolidating Commercial Workloads. This work appears in Proceedings of the 18th Large Installation System Administration Conference (LISA ’04).

Please note that while I (and my employer) retain the rights to electronically post this paper, you may not take a copy of it and repost it, according to USENIX rules. We’ll also work on getting this posted on BigAdmin.

On a personal note, I want to say “thanks” to Andy, whose idea this paper was in the first place, and without whose guidance would not have been accepted. And while this was fun, I’m tired and glad to be finished.

Zones goes to school

So I’ve been blogging all week about our efforts to teach people within Sun and at LISA about Zones (a.k.a. Solaris Containers). One of the attendees from Penn State came up to me just before my talk today and told me that an OS course at Penn State was looking at Zones as part of the course. I think I must have looked surprised. Now that I’ve had the chance to spend some time hunting through PSU’s CSE department website, I found that Prof. Don Heller’s CSE 411 course is working on this assignment. Hopefully our LISA ’04 paper will be of value to the students. I’ll pass a copy along (I hope to post the paper here, too, but I want to check in with the USENIX folks first). I think the professor has asked an interesting question, and that this sort of exercise (wade through various bits of literature trying to build an understanding of a topic) is valuable.

Being a Pennsylvania native, I’m particularly gratified 🙂

LISA ’04 Thursday: Booths and BoFs

Photo Credit: Jim Grisanzio
Another busy day! Today we spent more time at the Solaris 10 Booth at LISA ’04. I feel like we were effective at getting the message across. I also had a series of in-depth conversations with admins, IHVs, and others. A lot of people seem interested in our demos. One thing we’ve learned at LISA is that some of our users are cranky about Solaris patching. One told me it was forcing him off of Solaris. Several folks stopped and it was on their mind that their system uptime is impacted by slow patching, and that they prefer solutions such as that from debian. On the other hand, some people were appreciative of the ability to backout a patch. We talked a little bit about some remedies in the works– but I’m going to have to do some more research before I write it up.

Krister and I also went to a couple of the technical talks, and then to the Sci-Fi/Casino/Reception, where we had a great discussion about zones and jails implementation, and Open Source Solaris details with some of the guys from DragonFly BSD.

Capping the day, we hosted a BoF entitled Solaris 10: Where we are, and where we are going. I think it’s fair to call it a marathon session; I started by presenting an overview of Solaris 10, and then Andy talked about the OpenSolaris project. After about 80 minutes of presentation, we opened the floor up to discussion and questions, which was a really intense experience. We finished just a few minutes short of four hours, and repaired to the hotel bar for more discussion with the remaining folks– and then we got booted from there about a half hour later. Jim has pictures…

Alan took a bunch of notes at the BoF about feature ideas and suggestions from the community members present. I’ll see if we can get a summary posted soon.