Smacking Super-Smack into Shape for Solaris

A recent article
(and part 2) documented
the author’s attempts to benchmark MySQL performance on a
variety of operating systems. He noted
that a popular MySQL benchmark called
Super-Smack
doesn’t compile on Solaris. I hate when this happens– inevitably
this tells the reader that Solaris isn’t really a serious platform for MySQL (how could it
be, if the benchmark doesn’t work?). But from the other benchmarks the author provides, we can see
that Solaris performs respectably; and I suspect that MySQL has itself not received as much tuning
under Solaris as, for example, under Linux. With the help of performance
analysis tools like DTrace, trapstat, etc. we (or you, gentle reader) can fix that.

First, I’d like to clarify one claim in the article: Solaris 10 is bundled with a compiler. That
wasn’t true in the beta build the author used; but it is true as of the FCS build. So, the
benchmark will compile without installing any additional software.

I decided that it was time to get Super-Smack working under Solaris. The first task was to
get the program to compile.
The configure script ran OK, and I elected to just use the MySQL included in Solaris 10.
For more serious benchmarking, I would study this to decide whether to build MySQL myself. So:

$ PATH=/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin
$ export PATH
$ ./configure --with-mysql \\
--with-mysql-lib=/usr/sfw/lib/ \\
--with-mysql-include=/usr/sfw/include/mysql \\
--prefix=/home/dp/super-smack

I then had to make a few edits: src/Makefile needs to link the benchmark with the
additional libraries -lsocket -lnsl. A proper autoconf setup should detect this, but… no problem. A few minor edits to C files were also needed:

  • Added #include <strings.h> to engines.cc for bzero.
  • In query.cc, replaced calls to flock() with calls to fcntl(3c):
    -  flock(1, LOCK_EX);
    +  fcntl(1, F_SETLK, F_WRLCK);
    

It turns out that flock is used only sparingly, at the end of the benchmark
run, so we don’t need to pay attention to any performance implications.

So now, it built cleanly! Hooray. Next, I muddled my way through getting mysqld started.
Once I did, I had to cope with one more problem: Super-Smack, by way of libmysqlclient,
seems to want to access the mysql database via a UNIX domain socket at /var/lib/mysql/mysql.sock. However, the database seems to put that socket in
/tmp/mysql.sock. I wasn’t sure why, and I decided to investigate that discrepancy
out later. I hacked things up by putting an appropriate symlink in /var/lib/mysql to
work around the problem.

Next, I ran Super-Smack as instructed in the article, and things went somewhat haywire.
A quick look revealed that Super-Smack has a fairly conventional design: A parent process
forks a bunch of children, which do benchmark activities. When these are finished,
they write information back to the parent. I received a variety of error messages, and
after applying truss to some abbreviated runs, Jonathan and I decided that the parent
super-smack process was exiting prematurely, and failing to collect the data being
sent to it by its children. A quick scan of the source code led
me to this innocuous looking line of code:

pid_t pid = wait4(-1, 0, 0, NULL);

This is where the master super-smack process waits for its children.
My brokenness-sense was tingling. That -1 just looks wrong. And,
for Solaris, it is. This first argument, -1, is the pid to wait for.
In Linux’s wait4, this is implemented as follows (excerpted from the
linux man page):

< -1
which means to wait for any child process whose process group
ID is equal to the absolute value of pid.
-1
which means to wait for any child process; this is equivalent
to calling wait3. 
0which means to wait for any child process whose process group ID
is equal to that of the calling process.
> 0
which means to wait for the child whose process ID is equal
to the value of pid.

So now we know what the author meant: Wait for any child process.
While not fully documented (which is a bug), Solaris implements a slightly
different ruleset:

< 0
which means to wait for any child process whose process group
ID is equal to the absolute value of pid. 
0which means to wait for any child process; this is equivalent
to calling wait3.
> 0
which means to wait for the child whose process ID is equal
to the value of pid.

So, on Solaris, wait4(-1, …) instructs the OS to wait for any child
process whose process group ID is 1, while on Linux, it does a
wait3(). damn. A final note is that wait4()
is not defined by POSIX, the Single Unix Spec, or any standards body I
could find. Please, write portable code! One wonders why wait3()
wasn’t used in the first place. Quickly changing the code fixes the
problem.

At this point, I have what appears to be a working Super-Smack on Solaris,
and some initial results. I’ll intentionally not mention what hardware
this was run on, since I’ve not bothered to perform even rudimentary performance
analysis:

$ super-smack /smacks/select-key.smack 10 10000
Query Barrel Report for client smacker1
connect: max=330ms  min=6ms avg= 59ms from 10 clients
Query_type      num_queries     max_time        min_time        q_per_s
select_index    200000          0               0               8590.39

mpstat(1m) shows that this benchmarks spends a lot of time abusing
the system call path, and twiddling bits in userland; it’s not clear whether
this is really a good test of MySQL performance, since the test client and the
database have to fight for CPU resources…
All in all, not a terrible night’s work. I owe Jonathan a big thanks for
his help!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s