MySQL Proxy has a c-based event driven core that has command line support, event handling and a Lua-based scripting layer. The proxy is loading into MySQL via the plugin interface. With the community edition, you are responsible for writing your own scripts. But with MySQL Enterprise, it will come with several pre-built scripts that add some enhanced functionality.

The most basic use of MySQL Proxy is to rewrite or alter your queries. If you have a costly query that is killing your db, but you’re not sure where it is coming from, you can write a Lua script to either reject the query or to alter it and make it more manageable. It stops the dba from having to wait for a developer to update code and/or deploy the fix.

Another trick you can do with MySQL Proxy is to create new SQL commands. While this is probably not going to be a great idea, you can see a few interesting examples at http://thenoyes.com/littlenoise/?p=63. It is a good way to expand on functionality at the lowest level. Or to lose/forget where you put it when you switch to another db.

One of the features MySQL was thinking of when they built MySQL Proxy is the ability to manage your queries at the database level. Imagine tracking query quotas, by ip, user, db, etc., then rejecting queries automatically if over quota. Or perhaps to simulate a db outage or transaction failure when testing code to ensure properly fail-over or error handling.

One of the default Lua scripts that comes with MySQL Proxy is a basic load balancer for the db. It rotates the connections round-robin betweeen the MySQL servers. Since it is in the database layer, it can be aware of slag lag and adjust the rotation accordingly. Since it is a Lua script, you can customize it to work how you want it to.

In MySQL Enterprise, this script is on steroids and does so much more. The load balancer automatically checks the slave state, including lag, # of connections, last error, etc., and kicks the server from rotation if it finds something “wrong” with the slave. This functionality will be integrated with MySQL Enterprise Monitor so that you will have an easy view into how the load balancer is working.

Also for Enterprise, the MySQL team is working on an automated sharding algorithm to go with MySQL 5.1′s table partitioning. I’m not sure how easy that will be to implement, but if they are able to pull it off, I’ll be mighty happy. At least for basic partitioning or sharding, like with a user table, you could do it through MySQL Proxy without having to worry about missing an update someplace in your code. If you combine this with logging, you can update your code at your leisure and easily find the queries you missed without having to worry about breaking the site.

Other planned improvements include being able to run parallel queries, two-phased commits and syncronyous replication. Since the proxy is not the database engine, it can use multiple cores on a server and assign out queries and other processes in ways that the db engine you are using cannot.

This is still very new technology/product for MySQL, but I’m looking forward to what comes out of it. There is a lot of possibility in it, as well as the potential for bloat. Personally, I think it will end up back as MySQL core so that the engines can take advantage of some of the Proxy abilities.

Maria was designed to be a crash-safe replacement for MyISAM. Maria supports the same row formats and features as MyISAM. This means it can be fully compatible with MyISAM by simply running it in non-transactional control. It will be ACID compliant and have multi-version concurrency control (MVCC).

Version 1.0 of Maria was released in January as a branch of MySQL. It will continue to be a branch until version 2.0 where it will hopefully be part of MySQL 6.  Customized documentation is also available on http://dev.mysql.com/doc/refman/5.1-maria/en/se-maria.html.

Version 1.5 of Maria should be released any week now. Monty had originally hoped to announce it this week at the conference, but it is not quite done yet. He says that it should be out sometime between Late April to mid-May. Starting with version 1.5, Maria will support concurrent INSERT and SELECT operation. This will give Maria a nice feature merge between InnoDB and MyISAM.

Starting with MySQL 6 in Q4 of 2008, version 2 of Maria will be fully transactional and ACID compliant. Having the crash recovery will be a great thing to have with the read speed of MyISAM. Version 3 will be optimized for higher concurrency.

With the expected small footprint and the efficient data storage model, Maria will be a good engine of choice for desktop applications or embedding in applications.

The way in which Maria is crash-safe is kind of interesting. It stores everything except table definitions into log files. While these log files can be quite large, starting size is 1GB, it creates a few cool features. With MyISAM tables, you can apply a lock to the db (or shutdown the server) and then copy the data to a new server. With Maria, you can simply copy the data and the logs to the new server. Like with MyISAM, the new server will have corrupted tables. Unlike MyISAM, Maria will then run through the logs and recover the db to where it should be. Or, if you don’t like that method, you can take last night’s backup to the new server, copy just today’s logs from the donor server and start up MySQL. Maria will take the data from the backup and then automatically apply the logs until the new server is up-to-date.

There is quite a list of features that are either already in Maria or will be by the time version 3 comes out.

• MVCC & ACID
• Equal lookup speed for all indexes
• Allows easy copying of tables and/or Maria logs
• Has instant COUNT(*) queries
• Better BLOG handling than MyISAM
• Versioned
• Crash safe
• Data stored in 8k pages for less locking and improved query performance

By using 8k pages Maria will be faster on static size indexes than MyISAM, but slower on variable length keys. Updates that are planned for version 2 or 3 will make variable length indexes just as fast.

There are a few disadvantages of Maria over MyISAm.

• No INSERT DELAYED (may or may not eventually be added)
• Fulltext indexes are not yet crash-safe (planned to be fixed in version 3)
• No external locking
• Like MyISAM it will not support RAID
• When using checksums, the data size will be slightly larger for indexes
• Storage of very small rows (<25 bytes) are not efficient with the new page format

All in all, I’m pretty excited about where they are going with Maria. While some of the databases I use are write heavy the vaste majority are primarily reads; there are only so many news stories a day. I looks like Maria will give us the stability and locking abilities of InnoDB while keeping the speed of MyISAM reads.

The new table partitioning system has several benefits for your large datasets where you are having performance issues due to the volume of data. It can reduce the seek & scan set sizes and reduce the INSERT/UPDATE transaction durations. If you are already performing UNION queries or secondary queries to find data in an archive table, this functionality may be for you.

There are four (point 5) partitioning styles available for your use:

1. Range
   • Very useful for situations where the data is accessed by date
   • Usually used when a limited number of primary partitions are needed
   • It supports sub-partitions (the .5 partition type)
2. List
   • Good for when you are grouping data in partitions out of order (e.g., SELECT 1,5,7 in partition X)
   • As the name implies, you list which data goes into which partition
   • Higher maintenance costs in many situations
   • Supports sub-partitions
3. Hash
   • Usually has a lower maintenance cost
   • Partitions the data based on the hash that you tell it
   • Good for when you have non-intelligent keys
   • Works with both a limited and a large number of Partitions
4. Key
   • Similar to Hash partitioning, but based on one or more of the fields in the primary key instead of a user-defined hash
   • If there is no primary key, then the first unique index is used
   • Does not require the key to be integer based
5. Subpartitions
   • As the name implies, a partition within a partition
   • You can only create subpartitions for Range & List
   • If needed, subpartitions can be spread across volumes

The alter table statements are fairly simplistic; “ALTER TABLE my_table PARTITION BY KEY() PARTITIONS 50″ will create 50 partitions for that table.  If you need to add more partitions, you can just run “ALTER TABLE my_table ADD PARTITION PARTITIONS 25″ and your table will now have 75 partitions.  Of course you can shrink the size of the partitions as well.  One of the nice features of the partition modification is that the data is re-allocated as appropriate.

All in all, this looks like a much better solution than the standard 10/90 tables.

The home for the MySQL Sandbox project is hosted on SourceForge.net. It is not an officially supported product which is why Giuseppe keeps it on SourceForge.

The Sandbox was designed so that you don’t have to go through the hassle, and potentially error prone, steps of setting up multiple sandbox databases. It’s very common for these manual installations to accidentally install in the same data directory (can corrupt your data) or install with the same port/socket (MySQL won’t start).

Details of installing and using after the break…

The simple installation steps for MySQL Sandbox automatically creates a new instance of MySQL with a sandboxed data directory, port, and socket. In order to use Sandbox’s express_install script you need to download and un-tar a copy of MySQL for each version you wish to install. The script by default uses $HOME/opt/mysql as its base directory so be sure to extract MySQL there. You can change the default source directory, but by being in your home directory it helps keep your instance separate from everyone else’s.

Next download and extract the MySQL Sandbox from SourceForge and run the following:
./express_install.pl {directory in $HOME/opt/mysql}
If you prefer, you can run ./express_install.pl /path/to/mysql/directory

You can setup replication or multiple instances by ./set_replication.pl and ./set_custom_many.pl respectively. You also have a variety of install options that can be found by adding –help to the commands.

One of the nice commands you can use once the sandbox is setup and running is “clear.” This command will reset the data and instance back to default so that you can start again from scratch. Running the “use” command will get you into the CLI of the appropriate instance. One thing to keep in mind is that the commands to interact with the sandboxes must be run from the installed directory. You also have access to some of the standard MySQL command line tools by running “my {sqldump|sqlbinlog|sqladmin}” (Note the space after my).

It seems like this would be a good system to use in conjunction with DBUnit. As I talked about yesterday, DBUnit can auto-load/populate data. By putting the two together you should have a very maintainable and separate testing environment.

The new format of InnoDB can be dynamically installed w/o re-linking MySQL, which should be pretty handy for people who cannot easily re-build MySQL. The InnoDB upgrade is fully compatible with existing InnoDB tables and can even be downgraded back to the old version if you need to.

Installation is a breeze and can be done by downloading, extracting the ha_innodb.so into the MySQL Server plugin directory, and then restarting the mysqld. Be sure to enable “skip_innodb” in the my.cnf so that it doesn’t conflict with the engine already in MySQL. The innodb_file_format is now Barracuda. Finally log into the cli and “INSTALL PLUGIN INNODB SONAME ‘ha_innodb.so’” and “INSTALL PLUGIN INNODB_LOCKS SONAME ‘ha_innodb.s’”

MySQL/InnoDB 5.1 rebuilds the entire table row by row when creating a new index. The InnoDB Plugin builds just the new index. DROP INDEX for secondary indexes has improved performance as well because it only changes the data dictionary. When changing the PRIMARY KEY or FOREIGN KEYs it still needs to rebuild the table; it’s faster then 5.1.

Any C/U/D style queries will wait on the index creation, but SELECTs will still go through in shared mode. Newly created indexes lack the historical version info for row (for only that index). This can affect consistent read SELECTs of older transactions.

According the benchmarks that Peter Zaitsev performed, creating a new index on a 3GB table it takes 88 minutes with version 5.1 and an amazing 8 minutes with the new InnoDB plugin.

With the new table compression you can choose the compressed page size on a per table basis. In order to use this functionality, you need to have innodb_file_per_table = 1 and innodb_file_format = barracuda. The patterns in the data will determine your compression rate. According to Heikki, it can often be >50%. Both the tables and the indexes are compressed using zlib. It now trys to do updates and inserted in small sets so that it does not need to recompress when the data is updated. Deletions do not require page recreation.If the page is frequently accessed, it will keep it uncompressed. It is important to monitoring the compress vs uncompressed rates in the INFORMATION_SCHEMA. As few as 1% unsuccessful compression operations wastes your cpu time.

The best time to use compression is when you are I/O bound. If you are cpu-bound, then it is not a good idea to use compression as it will spend even more cpu time compressing and uncompressing. As a test, you can gzip your .ibd data file and see what level of compression you get. If it does not compress to significantly less than 50% of the original size, it probably won’t be worth doing.

The new ROW_FORMAT = DYNAMIC also requires the Baricuda file format. It is meant for situations when you have a table with a lot of large data elements like BLOB or TEXT. Now InnoDB can store the data off-page.

With the INFORMATION_SCHEMA updates, you can now find out much more information about the status of InnoDB and the individual tables. You can even now easily find the blocking queries and which process id is the owner. This should be a great help to DBAs who need to manage large sets of InnoDB databases.

After the keynotes, I decided to start off with the Testing PHP/MySQL Applications with PHPUnit/DbUnit session by Sebastian Bermann. I know I should already be using PHPUnit, but honestly I’ve never gotten around to it. I’m hoping that this will give me the kick in the ass that I need to start.

The first thing that Sebastian wanted us to note is that the PHPUnit listed on SourceForget.net is not the real PHPUnit. I kind of thought that weird and confusing. You’d think by now that he could get that old project de-listed.

Installing PHPUnit is fairly simple. It is not hosted off the standard PEAR channel, but by a quick extra command you can get access to it.
pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit

By setting up a reusable testing environment with PHPUnit you’ll be able to do automatic execution of your test code. This would be good for things like over night tests of the latest code commits. If a test fails, the developer team can be immediately notified. Be sure to enforce strict separation between production and testing environments. The last thing you want to happen is to have your test cases somehow break production.

Setting up a test case can be as simple or complex as you need it to be. You setup a class for each element/object you need to test. Each test case is a member function of this class where you assert value comparisons. While the documentation examples are best, here is a very simple example:

<?php
class BankAccountTest extends PHPUnit_Framework_TestCase {

    public function testBalanceIsInitiallyZero() {
        $ba = new BankAccount;
        $this->assertEquals(0, $ba->getBalance());
    }
}
?>

Where things can now start to get interesting is when you use the database extension to PHPUnit. The DBUnit extension was ported from jUnit by Michael Lively Jr.

DBUnit uses PDO to talk to a default SQLite database. However, a database it not required as it can emulate a database in memory or via XML. It also works with MySQL, but if the tests are written in a way that is compatible with SQLite, then the tests can be faster by about 20%.

Once of the nice things about how DBUnit works is how it puts the db into a known state between each test run. This means it avoids problems with one test case corrupting the db for other tests. By using its ability to import/export data to/from XML datasets, your db tests will run reliably and accurately.

Like with PHPUnit, DBUnit has assert methods like “assertTableEquals(expected, actual)” and can use PHP arrays for comparison. This can be good for comparing records in the database. e.g., The db gets updated via a PHPunit test and you then do a DBUnit test to verify the action.

A quick example of a DBUnit test case:

<?php
class BankAccountTest extends PHPUnit_Extensions_Database_TestCase {
    protected $pdo;

    public function __construct() {
        $this->pdo = PHPUnit_Util_PDO::factory($dsn);
        BankAccount::createTable($this->pdo);
    }

    protected function getConnection() {
        return $tihs->createDefaultDBConnection($this->pdo, 'mysql');
    }

    protect function getDataSet() {
        return $this->createFlatXMLDataSet('/path/to/seed.xml');
    }
}
?>

Brian Aker has put the PDF of the slides at http://download.tangent.org/talks/Memcached%20Study.pdf. The slides will be updated as time goes on so this link should always have the most up-to-date stuff. In case at some point he removes them, you can find them attached.

2:15pm: Talking about Grazr and how they have a daemon to do write-through cache. Now talking about processing data so that the data normally/frequently used is always in cache.

2:25pm: Memcached is supposed to be “simple” so that it can be faster. Has its own memory slab allocator, it originally tried using malloc but that was way too slow. The way it assigns blocks for memory assignment means that each data store goes into one of those “blocks.” If the # of available room in the block is full, then it drops the oldest record in that block. Written around libevent for scalable network connections. That was only connections sending data are “active” on the server.

2:27pm: The clients handle the majority of the load. It takes the cache key and hashes it so that it knows which server to send to the data to. Server doesn’t do the serialization either.

2:30pm: Server is not redundant and does not handle failover. But some clients like with PHP’s PECL, the client can implement this type of functionality.

2:33pm: Available commands are pretty much only set/get/replace/add, append/prepend, increment/decrement, cas and stats. While they say append/prepend are easy to be abused it is still an easy & quick way to store the keys used for something. Unless the size of the keys that you are storing reaches close to 1MB, it’s okay. It’s a great way to know what keys you’ll need to clear when product information has been updated.

2:38pm: MySQL UDF usage for Memcached is growing. Google Summer of Code is working on making MySQL Query cache use Memcached. lighthttpd has mod_memcache for caching files from disk. Apache also has mod_memcached but is still alpha.

2:42pm: Memcached has a few limits that you should pay attention to:
The max cache key is 250 bytes
Max data size is 1MB
Maxbytes limits the item cache, not everything
Be careful because with 32bit machines, you can set too high of a maxbytes that in combination with other memcache memory elements (e.g., key storage) and it will segfault.

2:46pm: LRU – Least recently accessed items are up for eviction and can be seen. One LRU exists per “slab class.” LRU evictions don’t need to be common. That’s pretty nice because you won’t lose larger data sets because a small data element doesn’t have room in its slab (and vice versa).

2:50pm: Threads – Great for large instances (16G+) and/or large multiget requests. It scales okay now, but they are working on improving it. Also means that you may not need to run multiple instances on the same box. Only 1 thread can talk to the allocator and the hash table at one time. This is so that you don’t have race conditions.

2:55pm: Don’t run Memcached with swap enabled or at least set it really small. Can seriously slow down performance and technically is contrary to purpose of memcache. The smaller swap means that OS can still use if it really has to, but won’t let the writing to swap happen for Memcached. The memory for Memcached is permanently allocated from the OS. Shouldn’t be an issue with most modern servers. The slab class are created by chunk size. Tends to create 36-39 slab classes. It does not reassign slab classes once the daemon loads. They are working on a way to allow you to change the assignments on the fly. e.g., if you find that you are evicting a lot of data from one slab, you can give that slab more pages while taking it from another slab.

3:10pm: There is normal hashing (usually crc or some modulus operation) and consistent hashing. Each client could implement its own version, but usually use common methods so that multiple clients can use the same pool of memcache servers.

3:20pm: PECL client (and most others) has option to not “remove” the server if it is not available. It can have a “back-off” method where it won’t try to hit the server for 1 second, 5 seconds, then 15 seconds, etc. It can also failover to a different server until the original is back online.

3:25pm: You should always try to use multi-get. Memcached is optimized for handling multiple requests at once. You’ll find a big improvement if you do this. The trick is to write your code in a way that can utilize this to the fullest.

4:04pm: Back from break. We’re now going over various coding examples of how to use Memcached. I’m hoping this part will be helpful. Most coding examples of how to use Memcached are generally fairly simple. But with 90 minutes left, there aught to be some gems.

4:15pm: Going over locks. Much like my previous post regarding memcached cache locks.

4:25pm: Just re-hashing the same examples in different languages.

4:35pm: libmemcached is a C/C++ client. Has replicated ability. You can store flags associated with the data in the byte category. Most clients don’t allow you direct access to this.

4:48pm: When doing a multi-get it hashes each key to determine the servers where the data would be stored. Once it knows that, the requests are made in parallel and as single-combined request to each server needed.

4:50pm: MySQL & Memcached. The MySQL uses the UDF API and libmemcached. You have to install it by CREATE FUNCTION. Most common method is to use memc_delete to remove the memcached data when data is written to the db. An example is “select id, url, memc_set(concat(‘feeds’, md5(url), url) from feeds;” or “select memc_get(concat(‘feeds’, md5(url));” This could be helpful for when you store the entire row, and only the row, in memcache. But for the most part, I don’t see the benefit of using this. Of course, I’ve been known to be wrong before.

5:00pm: There is memcached-tool which may assist with some basic stats/display commands. It will eventually be what does the slab re-allocation. libmmcached has memslap which may be useful for performance testing. MRTG gives you decent graphs about what is happening on the server.

5:05pm: 1.2.5 release supports multi-interface support, UDP all the time, noreply, & IPV6. Noreply is kind of neat as it allows you to set keys and not have have a response sent back to the client. It makes it a fire & forget save to memcache. My experience is that most people don’t verify that the data is really set to memcache anyway.

I really wanted to do the MySQL Proxy session, but that is an all-day thing and I’d miss the Memcached session. In hindsight, I’m probably going to regret that decision, but we’ll see. I can always do Proxy next year.

10:00am: So far pretty much all refresher. Some people have asked some good questions about the relay logs and replication lag. Unfortunately, none of it is particularly helpful. I don’t understand why MySQL doesn’t enable a way to do master-master replication reliably outside of NDB. In my opinion, NDB, is not an appropriate solution for most websites. Having to take down the entire cluster to alter a table would not be acceptable.

10:45am: Not sure if it is my laptop or the wireless network, but my connection/authorization never seems to carry over from when I log in/out of the laptop. I have to disconnect from the wireless auto-reconnect and connect again manually in order to get on the Internet. I wonder if anyone else is having this problem?

10:45am: One of the recommend ways for doing HA master/master is to use a shared disk array. The section would be to put a virtual IP in front of a heartbeat monitor in front of two MySQL servers configured the same. If they shared the same disk array, then the bin-log will be the same for both servers. The heartbeat monitor then sends the traffic to mas1 until it finds that it is no longer online. Then the traffic goes to mas2 automatically. When mas1 comes back online, the heartbeat monitor sends the traffic back automatically. Since mas2 is no longer reading/writing to the db tables, there shouldn’t be any corruption. Behind the mas1/mas2 is another virtual IP tied to the same heartbeat monitor. That way the slaves will also stay online. This sounds like it will handle many of the scenarios where you need HA master/master. The only thing I still see missing is the ability to alter tables without bringing down the tier like you can with m/cluster. [ m/cluster is a product by Continuent for HA synchronous replication. Of course, that product is EOL and their new product doesn't allow this either. ]

11:00am: Federated databases are good for joining two sets of data to another db server. I guess I always misunderstood how that works. This seems like it’d be a good way to hook up user information with community (forums) data. I wonder what the performance costs are?

11:15am: Row-based replication in 5.1 paves the way for future replication enhancements such as conflict detection & conflict resolution. As well as multi-channel replication and horizontal partioning. Multi-channel replication would be really cool. It is such a pain when replication lags and this could go a long way to help fight that. You cannot do master filtering for individual tables with row-based. I wonder why that is?

12:00pm: I don’t know why MySQL keeps pushing NDB as the end-all solution to replication problems. Yes it can deal with a lot of the replication issues, but it has several other issues that make it a bad (or at least not optimal) solution.

12:05pm: eek, running out of battery power. I have 18 minutes left… Should be able to make it until lunch. Maybe…

12:20: That’s about it. Q&A now, but I’m headed out to find a lunch table near power.