Generating data with dbmonster

fernando — Fri, 14 Aug 2009 20:02:56 +0000

In my last post I included some sample data which was useful for playing around with queries (once I published it,
I realized it made my post look like some form of keyword stuffing, fortunately I don’t use adsense on my site so I hope I’m free of any suspicion ).

That sample data was generated with dbmonster, a nice tool I discovered recently which comes in handy when you need to populate a database table to test your queries, the engine, schema, etc.

In a nutshell, here are the strengths I like about it:

It’s written in Java, so it’ll feed any database with a JDBC driver, and AFAIK, that’s just about any RDBMS you might need to use
It’s not too slow (it’s actually pretty fast if you don’t go crazy on long text based datatypes)
It includes several data generators, and being written in Java, rolling out my own should be easy.

As for the weaknesses:

It can be unbearably slow with long text columns
The generated text data might not represent a very real installation (again, look at my previous post for a quick sample). Still, the fact that you can easily write and plug your own generator(s) is enough to give kudos to the developers for a good architecture. Even if a feature isn’t available now, it’s painless to add in the future
~~It has no way to manage relational integrity, hence, it doesn’t really feed RDBMS, just JDBC enabled databases~~ (yes it does, please see the comments)

My last point means that, if you have table A and table B, both related through a foreign key constraint, dbmonster doesn’t provide a way for you to generate keys for table A and then reuse these keys
while inserting the foreign keys in table B. Not that I’ve found so far, at least.

Related with this limitation, tables are generated one at the time. Still, if you need a tool to populate tables, even very large ones, with random values, heck, up until a few days ago, I was doing this from
bash or python, so it’s definitely an improvement. I’ve discussed this with colleagues and some proprietary tools are either available or in the works that would handle referential integrity while
generating data for database testing. If anyone knows an Open Source one that does, it’d be great. Otherwise, that would sure make one hell of a good project

Ok, so enough rant, here’s some data: You get it from here.

Start off by creating a dbmonster.properties file. Here’s a sample:

dbmonster.jdbc.driver=com.mysql.jdbc.Driver
dbmonster.jdbc.url=jdbc:mysql://localhost/sample_db
dbmonster.jdbc.username=mysql_user
dbmonster.jdbc.password=mysql_password

# for Oracle and other schema enabled databases
#dbmonster.jdbc.schema=schema_name

# maximal number of (re)tries
dbmonster.max-tries=50

# default rows number for SchemaGrabber
dbmonster.rows=1000

# progres monitor class
dbmonster.progress.monitor=pl.kernelpanic.dbmonster.ProgressMonitorAdapter

Dbmonster is now ready to reach your database.
Before it’s ready to feed it, though, you need to create an xml file (ah, yes, the plague of the Java world, and I’ve been a Java guy for most of my working years…). Call it what you want,
being a consistent kind type of man, I’ve been using .xml. Here’s my sample_db.xml file:




  Sample DB

The file is pretty self explanatory. However, here are some pointers:

You have a table tag for every generated table
A column tag for every column of every table (duh)
Column’s have generators. These are the Java classes that actually generate the value to insert into this column. Notice how you can specify the databasedefault=”true” attribute, which makes dbmonster omit generation for this column (good for auto increment columns / postgres sequences

Now let’s run it:

export CLASSPATH=dbmonster*.jar:mysql*jar

java pl.kernelpanic.dbmonster.Launcher -s sample_db.xml

fipar@telecaster:~/soft/dbmonster-core-1.0.3$ java -classpath mysql-connector-java-5.1.6-bin.jar:dbmonster-core-1.0.3.jar pl.kernelpanic.dbmonster.Launcher -s sample_db.xml
2009-08-05 13:43:27,244 INFO  DBMonster - Let's feed this hungry database.
2009-08-05 13:43:27,697 INFO  DBCPConnectionProvider - Today we are feeding: MySQL 5.0.75-0ubuntu10.2-log
2009-08-05 13:43:27,783 INFO  Schema - Generating schema .
2009-08-05 13:43:27,783 INFO  Table - Generating table .
2009-08-05 13:44:18,749 INFO  Table - Generation of table  finished.
2009-08-05 13:44:18,750 INFO  Schema - Generation of schema  finished.
2009-08-05 13:44:18,750 INFO  DBMonster - Finished in 51 sec. 507 ms.
fipar@telecaster:~/soft/dbmonster-core-1.0.3$ mysql -p -e 'select count(*) from tasks' sample_db
Enter password:
+----------+
| count(*) |
+----------+
|    80000 |
+----------+

fipar@telecaster:~/soft/dbmonster-core-1.0.3$ java -cp dbmonster-core-1.0.3.jar:mysql-connector-java-5.1.6-bin.jar pl.kernelpanic.dbmonster.Launcher -s sample_db.xml
2009-08-06 10:27:24,747 INFO  DBMonster - Let's feed this hungry database.
2009-08-06 10:27:25,336 INFO  DBCPConnectionProvider - Today we are feeding: MySQL 5.0.75-0ubuntu10.2-log
2009-08-06 10:27:25,490 INFO  Schema - Generating schema .
2009-08-06 10:27:25,490 INFO  Table - Generating table .
2009-08-06 10:37:14,284 INFO  Table - Generation of table  finished.
2009-08-06 10:37:14,284 INFO  Schema - Generation of schema  finished.
2009-08-06 10:37:14,285 INFO  DBMonster - Finished in 9 min. 49 sec. 539 ms.
fipar@telecaster:~/soft/dbmonster-core-1.0.3$ mysql -p -e 'select count(*) from tasks' sample_db
Enter password:
+----------+
| count(*) |
+----------+
|   150000 |
+----------+

Besides the obvious difference in the number of rows, the second generation had larger text values (char(250) vs char(40)). You can see how that affected generation time. It should have finished (proportionally) in
about 1:50, yet it took dbmonser almost 10 minutes to generate this data.

In conclusion, depending on the size of the database you need to generate, and the use you intend for it (forget testing a huge schema with referential constraints), dbmonster can certainly aid
you in stress testing any database engine.

Generating random salts from bash From the ‘just because it can be done’ column, here...
Using the ENUM data type to increase performance While going through the DATA TYPES section of the Certification...
Piping data to multiple processes Here’s a simple shell script to stream data to multiple...

Related posts brought to you by Yet Another Related Posts Plugin.

Using MySQL sandbox for testing

fernando — Thu, 02 Apr 2009 23:28:05 +0000

MySQL Sandbox is a great tool for quickly deploying test MySQL instances, particularly if your daily work involves diagnosing problems across multiple MySQL versions.

Once you’ve downloaded it, it needs no installation. Just have a few MySQL binary releases at hand, and begin creating sandboxes in just a few seconds:

./make_sandbox mysql-5.0.77-linux-x86_64-glibc23.tar.gz

./make_sandbox mysql-5.1.32-linux-x86_64-glibc23.tar.gz

Pretty simple, huh?

Suppose you have a parallel build around, say, 5.0.77-percona-highperf. The default syntax won’t work if you’ve already created a 5.0.77 sandbox, since Sandbox will use ’5.0.77′ as the sandbox dir. In this case, you’ll need to manually specify a directory name:

./make_sandbox mysql-5.0.77-percona-highperf-b13.tar.gz -d msb_percona_5_0_77

Fortunately, all Sandbox commands have a –help option that provide helpful screens to let you know how you can modify their default behavior. In this particular case, you’ll actually need to access the help for low_level_make_sandbox, since that’s the script that’s eventually called.

Once a sandbox is created, using it is real simple. Just cd to it’s directory (by default, $HOME/sandboxes/$SANDBOXDIR) and issue one of the following commands:

./start
./stop
./restart
./use (this invokes the sandbox’s mysql client with appropriate parameters. you can append your own parameters too, i.e., a database name)
./clear (stops the server and removes everything from the data directory, leaving you ready to start from scratch)

It even includes an option to create replication sandboxes, with a master node and two slaves. This is by default, but you can also create multiple sandboxes of the same version and configure replication yourself.

Here’s an example:

./make_replication_sandbox mysql-5.0.77-linux-x86_64-glibc23.tar.gz

So there it is, once again, download it and start having fun!

Using MySQL Proxy to benchmark query performance By transparently sitting between client and server on each request,...
Indexing text columns in MySQL This time, I’m talking about indexes for string typed columns....
Updated mysql-proxy benchmarking script (for proxy 0.7) My previous post contained a lua script for MySQL proxy...

Related posts brought to you by Yet Another Related Posts Plugin.

Fernando Ipar » Testing

Generating data with dbmonster

Using MySQL sandbox for testing